Kafka

60-Day Kafka 4 Learning Plan · Week 8 — Day 54 of 60

Editorial Team

60-Day Kafka 4 Learning Plan · Week 8 — Production & Cloud Sources: Kafka: The Definitive Guide Ch.8 · kafka.apache.org/documentation/#georeplication

Goal

Set up MirrorMaker 2 (MM2) to replicate Kafka topics between two clusters, understand active-passive and active-active patterns, configure offset synchronisation for seamless failover, fix a real gap in the active-passive setup around topic naming that breaks consumers exactly when DR matters most, and monitor replication lag.

1. What is MirrorMaker 2?

MirrorMaker 2 replicates topics between Kafka clusters using Kafka Connect internally. It copies:

  • Records — all messages from selected topics
  • Consumer group offsets — so consumers can resume on the target cluster
  • Topic configurations — partition count, retention, compaction settings
  • ACLs — optionally, access control rules

MM2 was introduced in Kafka 2.4 (KIP-382) and replaces the original MirrorMaker. It is built on Kafka Connect, making it restartable, monitorable, and horizontally scalable.

2. Active-passive vs active-active

Active-passive (DR) ★

Primary cluster (DC1) ──── MM2 ────► Backup cluster (DC2)
labs.events                          primary.labs.events
  • Primary handles 100% of traffic
  • Backup is a hot standby — consumers can fail over
  • One-way replication only
  • Use cases: geo-DR, backups, cluster migration

The primary.labs.events renaming shown here is exactly the gap covered in §8 — worth reading before treating this as DR-ready.

Active-active (multi-region)

Cluster A (us-east) ──── MM2 ────► Cluster B (eu-west)
labs.events                        a.labs.events

Cluster B (eu-west) ──── MM2 ────► Cluster A (us-east)
labs.events                        b.labs.events
  • Both clusters handle traffic from their region
  • MM2 replicates each cluster’s topics to the other
  • Topic prefix prevents replication cycles (a.b.labs.events is never re-replicated)
  • Use cases: low-latency multi-region serving, geo routing

3. mm2.properties — active-passive config

# mm2.properties — replicate labs.events from primary to backup

# Cluster aliases
clusters=primary, backup
primary.bootstrap.servers=primary-broker:9092
backup.bootstrap.servers=backup-broker:9092

# Replication flow: primary → backup only
primary->backup.enabled=true
backup->primary.enabled=false

# Topic filter — regex, replicate all labs.* topics
primary->backup.topics=labs\..*

# Heartbeat topics — used to measure replication lag
primary->backup.emit.heartbeats.enabled=true
primary->backup.heartbeats.topic.replication.factor=3

# Offset sync — lets consumer groups resume on backup after failover
primary->backup.sync.group.offsets.enabled=true
primary->backup.sync.group.offsets.interval.seconds=60

# Topic config sync — keep partition count, retention in sync
primary->backup.sync.topic.configs.enabled=true
primary->backup.sync.topic.acls.enabled=false   # disable if backup has different ACL setup

# Replication factor for internal MM2 topics on backup cluster
replication.factor=3

# MM2 worker settings
tasks.max=4                         # parallel tasks per connector
offset.storage.replication.factor=3
config.storage.replication.factor=3
status.storage.replication.factor=33

4. Active-active mm2.properties

# Active-active: both directions

clusters=dc1, dc2
dc1.bootstrap.servers=dc1-broker:9092
dc2.bootstrap.servers=dc2-broker:9092

# Both directions enabled
dc1->dc2.enabled=true
dc2->dc1.enabled=true

# Each cluster replicates its own local topics
dc1->dc2.topics=labs\..*
dc2->dc1.topics=labs\..*

# Offset sync in both directions
dc1->dc2.sync.group.offsets.enabled=true
dc2->dc1.sync.group.offsets.enabled=true

replication.factor=3

Cycle prevention: MM2 automatically prefixes replicated topics. A topic labs.events from dc1 appears as dc1.labs.events on dc2. MM2 never re-replicates already-prefixed topics — this breaks the cycle.

5. Start MirrorMaker 2

Mode 1 — Dedicated MM2 process (dev / simple setups)

# Run MM2 as a standalone process
connect-mirror-maker.sh mm2.properties

# Or with the older binary
kafka-mirror-maker.sh \
  --consumer.config primary-consumer.properties \
  --producer.config backup-producer.properties \
  --whitelist "labs\..*"

Mode 2 — Kafka Connect cluster (production)

# docker-compose.yml — MM2 as a Connect worker
mm2:
  image: confluentinc/cp-kafka-connect:7.6.0
  environment:
    CONNECT_BOOTSTRAP_SERVERS: backup-broker:9092
    CONNECT_REST_PORT: 8083
    CONNECT_GROUP_ID: mm2-group
    CONNECT_CONFIG_STORAGE_TOPIC: mm2-configs
    CONNECT_OFFSET_STORAGE_TOPIC: mm2-offsets
    CONNECT_STATUS_STORAGE_TOPIC: mm2-status
    CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
    CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
    CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
    CONNECT_KEY_CONVERTER: org.apache.kafka.connect.converters.ByteArrayConverter
    CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.converters.ByteArrayConverter
  volumes:
    - ./mm2.properties:/etc/kafka/mm2.properties
  command: /bin/connect-mirror-maker.sh /etc/kafka/mm2.properties
  ports:
    - "8083:8083"

The now-familiar REPLICATION_FACTOR: 1 pattern shows up again here — the same dev-only shortcut flagged repeatedly since Day 36 §4, this time on MM2’s own internal Connect topics. For a DR tool specifically, under-replicating its own config/offset/status topics is a particularly ironic place to skip this — set these to 3 in any real deployment.

Verify via Connect REST API

# List running connectors
curl http://localhost:8083/connectors | jq

# Check connector status
curl http://localhost:8083/connectors/MirrorSourceConnector/status | jq

# Expected: {"state":"RUNNING","worker_id":"mm2:8083"}

# Verify replicated topics appear on backup
kafka-topics.sh --bootstrap-server backup-broker:9092 --list | grep primary
# primary.labs.events
# primary.labs.notifications
# primary.heartbeats

6. Offset translation — surviving failover

When consumers fail over from primary to backup, their committed offsets on primary don’t directly match offsets on backup (records may have different offsets even for the same messages).

MM2 maintains a checkpoint topic (primary.checkpoints.internal) that maps primary offsets to backup offsets.

Manual offset reset after failover (simplest approach)

# Reset consumer group to latest on backup — safe if you can afford to miss some messages
kafka-consumer-groups.sh \
  --bootstrap-server backup-broker:9092 \
  --group labs-api-group \
  --reset-offsets \
  --to-latest \
  --all-topics \
  --execute

Programmatic offset translation (zero message loss)

// Use RemoteClusterUtils to translate offsets
Map<TopicPartition, OffsetAndMetadata> primaryOffsets = adminClient
    .listConsumerGroupOffsets("labs-api-group")
    .partitionsToOffsetAndMetadata()
    .get();

// Translate to backup cluster offsets via MM2 checkpoint topic
Map<TopicPartition, OffsetAndMetadata> backupOffsets =
    RemoteClusterUtils.translateOffsets(
        backupAdminClient.describeCluster(),
        "primary",          // source alias from mm2.properties
        "labs-api-group",
        Duration.ofSeconds(30)
    );

// Reset consumer group on backup to translated offsets
adminClient.alterConsumerGroupOffsets("labs-api-group", backupOffsets);

7. Monitoring MM2 replication lag

# Check consumer group lag for MM2's internal consumer (measures replication lag)
kafka-consumer-groups.sh \
  --bootstrap-server primary-broker:9092 \
  --describe \
  --group mm2-primary->backup-labs.events-0

# Key Prometheus metric (via kafka_exporter)
kafka_consumergroup_lag{consumergroup="mm2-primary->backup-labs.events-0"}

# MM2-specific metric: age of the newest record at the source not yet replicated
# kafka.mirror.maker:type=MirrorSourceConnector,name=replication-latency-ms-max

Alert rule

- alert: MM2ReplicationLagHigh
  expr: kafka_consumergroup_lag{consumergroup=~"mm2-.*"} > 10000
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "MirrorMaker 2 replication lag is high — backup cluster falling behind"

8. IdentityReplicationPolicy — the topic-renaming gap that breaks failover exactly when it matters

§2/§3’s active-passive setup replicates labs.events as primary.labs.events on the backup cluster — this default renaming is deliberate (it’s what makes cycle-prevention in active-active mode possible, §4), but it has a serious, easy-to-miss consequence for pure active-passive DR: a labs-api/labs-socket instance configured to consume from labs.events will find no such topic on the backup cluster after failover — only primary.labs.events exists there. Exactly the scenario where you need failover to work flawlessly (an actual disaster) is when this naming mismatch first gets discovered, unless it’s addressed ahead of time.

# For pure active-passive DR (no active-active, no cycle risk to prevent),
# use IdentityReplicationPolicy to preserve topic names across clusters
replication.policy.class=org.apache.kafka.connect.mirror.IdentityReplicationPolicy
With IdentityReplicationPolicy:
  labs.events on primary  →  labs.events on backup  (same name, no prefix)

Consumers configured for "labs.events" work unchanged after failover —
no application config change needed at the exact moment of a disaster

The trade-off this creates: IdentityReplicationPolicy is safe specifically because active-passive is one-directional (§3’s backup->primary.enabled=false) — there’s no cycle to prevent since replication never flows back. It is not safe for active-active (§4), where the whole point of the default prefixing policy is preventing infinite replication loops between two clusters that both replicate to each other. Choose the replication policy based on which pattern you’re actually running, and if this is genuinely one-way DR, prefer IdentityReplicationPolicy specifically so failover doesn’t also require a coordinated application reconfiguration during an active incident.

9. Schema Registry is not replicated by MM2

Everything in §1’s “what MM2 copies” list is Kafka-native — records, offsets, topic configs, ACLs. Schema Registry (Day 23–27) is a separate system with its own storage (the _schemas topic, Day 23 §8) and MM2 has no built-in awareness of it at all.

The practical failure mode: if labs-api produces Avro-encoded events (Week 4) that get replicated to the backup cluster via MM2, but the backup cluster’s Schema Registry never received the corresponding schema registrations, any consumer on the backup cluster attempting to deserialize those Avro messages after failover will fail — the schema ID embedded in each message (Day 22 §2’s wire format) has no meaning on a Schema Registry instance that never saw it registered. Schema Registry replication needs to be handled as an explicitly separate concern: either run Confluent’s Schema Linking (Schema Registry’s own analog to MM2, mirroring subjects between Registry instances) or a custom process that keeps both Registry instances’ subjects in sync. This is very easy to miss when MM2 is set up and topic replication is verified working (§5), because the messages replicate successfully and only fail to deserialize later, on the backup cluster, at the worst possible moment.

10. MM2 delivery semantics — at-least-once, not exactly-once

MM2 is, at its core, a Kafka Connect source connector reading from the primary and producing to the backup (Day 36’s Connect model) — which means it inherits Connect’s default at-least-once delivery semantics (Day 36 §10) unless explicitly configured otherwise.

What this means in practice: a network blip or MM2 task restart during replication can produce duplicate records on the backup cluster for the same primary-cluster message — not data loss, but potential duplication. For most DR use cases this is an acceptable trade-off (better to have an occasional duplicate on failover than a gap), but it’s worth knowing explicitly rather than assuming replication is exactly-once by default. If exactly-once replication genuinely matters for a specific pipeline, MM2 does support exactly.once.source.support at the Connect worker level (the same mechanism from Day 36 §10, applied here) — but confirm it’s actually enabled rather than assuming it, and understand it adds real overhead (transactional writes, Day 11) that may not be justified for every replicated topic.

11. Testing failover procedures — a MirrorMaker 2 config is not a tested DR plan

Everything in §3-§8 sets up the mechanism for failover — it does not verify that failover actually works end-to-end, including the application-level reconfiguration (or lack thereof, with §8’s fix) that a real incident would require.

#!/bin/bash
# dr-failover-drill.sh — a periodic "game day" exercise, not a one-time setup verification

# 1. Confirm current replication lag is healthy before starting
kafka-consumer-groups.sh --bootstrap-server primary-broker:9092 \
  --describe --group mm2-primary->backup-labs.events-0

# 2. Stop producing to primary (simulating primary cluster unavailability)
# 3. Point labs-api/labs-socket at backup-broker:9092 (or verify IdentityReplicationPolicy
#    means no config change is needed at all — the actual point of §8's fix)
# 4. Verify consumers resume from the correct offset (§6) with no unexpected gap or duplication
# 5. Verify Avro deserialization succeeds (§9's Schema Registry gap, if applicable)
# 6. Document actual time-to-recovery — this becomes the empirical RTO for Day 55

Why this deserves to be a recurring exercise, not a one-time validation: a DR setup that was correctly tested six months ago can silently break from any of several ordinary changes since then — a new topic added without updating the MM2 topic filter regex (§3’s labs\..* needs to actually match every topic that matters), a Schema Registry subject added on primary but never replicated (§9), or an application config drift that reintroduces the exact naming assumption §8 fixed. Treat DR failover testing the same way Day 21 §5 treated integration testing — something that must actually run and pass, not something assumed to work because the configuration looks right on paper.

12. Common pitfalls

  • Deploying active-passive DR with default topic prefixing and no IdentityReplicationPolicy — consumers break at the exact moment failover is needed, unless application config was separately updated to expect the primary. prefix (§8)
  • Assuming MM2 replicates everything the application depends on — Schema Registry is a separate system MM2 has no awareness of; Avro/Protobuf consumers can fail to deserialize on the backup cluster even though message replication itself succeeded (§9)
  • Assuming MM2 replication is exactly-once by default — it’s at-least-once like any Connect source connector unless exactly.once.source.support is explicitly configured (§10)
  • Treating a working MM2 config as a tested DR plan — replication mechanics being correct doesn’t verify the actual failover procedure, application reconfiguration, or Schema Registry state works end-to-end (§11)
  • Leaving MM2’s own internal Connect topics at RF=1 — under-replicating the DR tool’s own configuration/offset tracking is a particularly poor place to skip this (§5)

Key Takeaways

  • MM2 is built on Kafka Connect — reliable, scalable, and observable like any connector, and inherits Connect’s at-least-once delivery semantics by default
  • Active-passive: one-way replication for DR — simplest and most common pattern
  • Active-active: bidirectional — topic prefix (dc1.) prevents replication cycles
  • Default topic prefixing (labs.events → primary.labs.events) breaks consumers expecting unchanged topic names after failover — use IdentityReplicationPolicy for pure active-passive DR to avoid this
  • MM2 does NOT replicate Schema Registry — Avro/Protobuf consumers can fail to deserialize on the backup cluster after failover unless Schema Registry is separately kept in sync
  • sync.group.offsets=true lets consumers resume on backup without losing position, but exact offset translation (§6) still matters for zero-loss failover
  • Monitor lag between clusters with kafka_consumergroup_lag for MM2’s consumer group
  • A correct MM2 configuration is not a tested DR plan — run periodic failover drills that exercise the full application-level recovery path, not just replication mechanics

Support me through GitHub Sponsors.

Thank you for Reading !! See you in the next post.

Next

➡️ Day 55: Disaster recovery — failover, backups, runbooks

Resources

Link to Medium blog

, ,

Related Posts

Kafka

60-Day Kafka 4 Learning Plan · Week 9 — Day 60 of 60

Editorial Team
Kafka

60-Day Kafka 4 Learning Plan · Week 9 — Day 59 of 60

Editorial Team