MirrorMaker 2.0 Consumer Group Offset Syncing Lag in Active-Passive DR Setup

[prth]

We are using Strimzi’s Kafka MirrorMaker 2.0 to replicate messages and consumer group offsets between two regions in an active-passive (DR) setup. While messages sync correctly, we observe significant lag in consumer group offset syncing.

Issue Description
• Message replication is working fine, with no noticeable lag.
• Consumer group offsets are not syncing in a timely manner.

Here is configuration used for source and checkpoint connector

     sourceConnector:
        tasksMax: 4
        config:
          replication.factor: -1
          offset-syncs.topic.replication.factor: -1
          refresh.topics.interval.seconds: 600
          replication.policy.class: "org.apache.kafka.connect.mirror.IdentityReplicationPolicy" 
          consumer.auto.offset.reset: earliest 
          offset.lag.max: 5
          sync.topic.acls.enabled: false
      checkpointConnector:
        tasksMax: 4
        config:
          checkpoints.topic.replication.factor: -1
          sync.group.offsets.enabled: true
          refresh.groups.interval.seconds: 10
          replication.policy.class: "org.apache.kafka.connect.mirror.IdentityReplicationPolicy"
          emit.checkpoints.interval.seconds: 5
          sync.group.offsets.interval.seconds: 5
      topicsPattern: "^(?!mirrormaker2-).*"
      groupsPattern: ".*"

We tried tuning/reducing consumer group sync intervals (emit.checkpoints.interval.seconds, sync.group.offsets.interval.seconds), but the issue persists.
We also reduced offset.lag.max on the source connector, but it did not improve offset syncing.

Are there any configurations we can tweak to improve consumer offset syncing performance?

[scholzj]

I guess a good start would be:

• Proper explanation what significant lag means
• Full custom resource
• Full logs

[prth]

Here is the what we were using

# https://artifacthub.io/packages/helm/strimzi-kafka-operator/strimzi-kafka-operator
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: mm2-dr-setup-1
  namespace: kafka
spec:
  version: 3.8.0
  replicas: 2
  connectCluster: "targetRegion" # Must be the target custer
  clusters:
    - alias: "sourceRegion" # Source cluster
      bootstrapServers: {{source-server}}
      config:
        # -1 means it will use the default replication factor configured in the broker
        config.storage.replication.factor: -1
        offset.storage.replication.factor: -1
        status.storage.replication.factor: -1
    - alias: "targetRegion" # Target cluster
      bootstrapServers: {{target-server}}
      config:
        # -1 means it will use the default replication factor configured in the broker
        config.storage.replication.factor: -1
        offset.storage.replication.factor: -1
        status.storage.replication.factor: -1
  mirrors:
    - sourceCluster: "sourceRegion"
      targetCluster: "targetRegion"
      sourceConnector:
        tasksMax: 4
        config:
          replication.factor: -1
          offset-syncs.topic.replication.factor: -1
          refresh.topics.interval.seconds: 600
          replication.policy.class: "org.apache.kafka.connect.mirror.IdentityReplicationPolicy" 
          consumer.auto.offset.reset: earliest 
          offset.lag.max: 5
          sync.topic.acls.enabled: false
      checkpointConnector:
        tasksMax: 4
        config:
          checkpoints.topic.replication.factor: -1
          sync.group.offsets.enabled: true
          refresh.groups.interval.seconds: 10
          replication.policy.class: "org.apache.kafka.connect.mirror.IdentityReplicationPolicy"
          emit.checkpoints.interval.seconds: 5
          sync.group.offsets.interval.seconds: 5
      topicsPattern: ".*"
      groupsPattern: ".*"
  metricsConfig:
    type: jmxPrometheusExporter
    valueFrom:
      configMapKeyRef:
        name: mirror-maker-2-metrics
        key: metrics-config.yml
  resources:
    limits:
      cpu: 2
      memory: 4Gi
    requests:
      cpu: 2
      memory: 4Gi

We are testing consumer group offset sync across regions using MirrorMaker 2 with the checkpoint connector. The goal is to evaluate how quickly consumer offsets are mirrored in a disaster recovery (DR) scenario.

Test Scenario:

1. Message Production: In the source region, we pushed a bulk of ~50K messages into a topic.
2. Slow Consumer: A Spring Boot Java application was set up as a slow consumer in the source region, processing one message at a time with a 1-second delay per message. It also committed offsets one at a time.
3. Monitoring Offset Sync:
   • Used AKHQ to manually refresh and track consumer group offsets on the target region.
   • Developed a Kafka Admin API-based utility that checked and logged consumer group offset changes every 2 seconds. And note that I was tracking this consumer group offset change on topic partition level
4. Target Region: No active consumers; it remained in standby mode.

Observations --

• Message Sync: Messages replicated instantly from source to target.
• Offset Sync Delay: The slow consumer progressed in source region at its stable rate. But in the target region, the consumer group offsets won't sync quickly, it would change every 30-45 minutes.
• Offset Sync Behavior:
  • As the source consumer group’s offset reduced (e.g., around 1000 remaining), the sync frequency improved to once every 2-3 minutes.
  • When we stopped the consumer in the source region, we expected MirrorMaker to sync the final consumer group offset to the target. However, after stopping the source consumer, no further offset updates occurred, even after waiting for an hour.

Concern:
For disaster recovery, we need faster consumer group offset sync since a 45-minute delay is too long for failover scenarios. This becomes a major issue for DR drills, where we need to switch regions quickly.
Is there a way to improve the offset sync speed or force a faster sync when the source consumer stops? Any recommendations or known limitations regarding this behavior?

[scholzj]

I think that is pretty much as it is expected to work. The sync happens in batches and the tracking of the past offsets might be limited for a consumer group that is far behind.

[prth]

Ohhh is that so… Thank you for replying and sharing this info! That's good to know in a way, as I was constantly wondering what have I missed.

Could you please explain why this behavior occurs?
I’m a bit confused because the documentation mentions properties like sync.group.offsets.interval.seconds and emit.checkpoints.interval.seconds (Here)

• As these were “interval” seconds, I was expecting a somewhat constant rate of sync. It gave me an impression that offset sync happens at fixed intervals.
  I couldn't find any documentation or explanation that the sync frequency depends on the consumer group’s lag or state.

--

And few more questions please --

1. If a consumer group having a certain non-zero offset is inactive in the source cluster, does that mean its offset won’t be synced to the target region?
2. Is there a way where we can force a consumer group offset sync like through some API call, which we can leverage during performing drills to switch to standby region.

[scholzj]

I'm not a MM2 expert to give you detailed answers. The Kafka community mailing lists would be the best place for it. But imagine it like this -> a source cluster might have billions of records each with its own offset. And when these records are mirrored to the target cluster, they would get each their own offset there. MM2 has no chance to have an exact make of which offset on the source cluster matches which offset on the target cluster. That would require huge amount of resources etc. So there have to be some limitations to what offsets can be translated (how far back), how often, after how many messages, etc.

If a consumer group having a certain non-zero offset is inactive in the source cluster, does that mean its offset won’t be synced to the target region?

I don't think the group being inactive in the source cluster should play any role.

Is there a way where we can force a consumer group offset sync like through some API call, which we can leverage during performing drills to switch to standby region.

No. Plus that would not be useful too much either I guess because in the case of disaster, you will get what is synced at the time of the disaster. So any API you can call before the switch would really help much in real life.

[prth]

Understood,

and about those other 2 questions --

1. About inactive consumers

I don't think the group being inactive in the source cluster should play any role.

This is an odd behaviour that I have observed, I will retry this scenario again and confirm.

---

2. About API...

No. Plus that would not be useful too much either I guess because in the case of disaster, you will get what is synced at the time of the disaster. So any API you can call before the switch would really help much in real life.

Yes, that is certainly true. It's just having said that, for doing "DR drills" to test our platform readiness, was hoping to avoid any issues caused by intentionally switching regions. Since, if the consumer offsets aren’t up to date, switching consumers to the standby region could result in some messages being reprocessed.

So, we were wondering—

• In our pipeline that triggers the region change, could we make an API call to force an offset sync?
• Or is there any other way to manually trigger offset synchronization?

(On side note - Initially, I assumed the sync was interval-based, so I tried estimating the time intervals. My plan for DR drills was to first stop the source consumer and producer services, then add an intentional wait for MM2 to catch up with message and offset sync. And, then stop MM2 process and start target region consumers and producers. But this approach won’t work.)

[scholzj]

There is no API call AFAIK. If you do a planned switch, you can simply move producers first and let the consumers finish at which point they should be closer to being in-sync (but even then, it might not be full yin sync as the sync always happens after batches of N records).