60-Day Kafka 4 Learning Plan · Week 8 — Day 55 of 60

60-Day Kafka 4 Learning Plan · Week 8 — Production & Cloud Sources: Kafka: The Definitive Guide Ch.8, 10 · kafka.apache.org/documentation/#basic_ops_restarting

Goal

Build a production-ready DR strategy: define RTO/RPO targets (distinguishing single-broker RPO from cluster-level DR RPO, which are governed by completely different mechanisms), understand Kafka failure modes, write actionable runbooks for broker and cluster failures — fixing a real missing step in the cluster failover runbook that connects directly to Day 54’s topic-naming lesson — and establish a tested backup regimen.

1. RTO and RPO — define targets first

Config for near-zero RPO:

# server.properties
default.replication.factor=3
min.insync.replicas=2

# producer config
acks=all
enable.idempotence=true
retries=Integer.MAX_VALUE

This table quietly conflates two different RPO mechanisms — see §8 for why that matters. “Near-zero with RF=3 + acks=all” is true for a single-broker failure within a healthy cluster; it says nothing about RPO when the entire primary cluster is lost, which is bounded by MM2 replication lag (Day 54 §7) instead.

2. Failure scenarios and recovery

3. Single broker failure — runbook

#!/bin/bash
# broker-failure.sh — executed by on-call engineer
# Prerequisites: admin.properties with bootstrap credentials

ADMIN_PROPS="/tmp/admin.properties"
BOOTSTRAP="broker-1:9092"

echo "=== Step 1: Confirm broker is unreachable ==="
kafka-broker-api-versions.sh \
  --bootstrap-server broker-2:9092 \
  --command-config $ADMIN_PROPS
# If this succeeds, cluster is still functional — proceed

echo "=== Step 2: Check under-replicated partitions ==="
kafka-topics.sh \
  --bootstrap-server $BOOTSTRAP \
  --describe \
  --under-replicated-partitions
# Expected: zero output once the failed broker is replaced

echo "=== Step 3: Restart or replace the failed broker ==="
# Option A: Kubernetes rolling restart
kubectl rollout restart statefulset/labs-kafka-broker -n kafka
kubectl rollout status statefulset/labs-kafka-broker -n kafka --timeout=300s

# Option B: Replace EC2 node and rejoin cluster
# (broker auto-rejoins using stored node ID and data directory)

echo "=== Step 4: Trigger preferred-replica election ==="
kafka-leader-election.sh \
  --bootstrap-server $BOOTSTRAP \
  --election-type preferred \
  --all-topic-partitions \
  --command-config $ADMIN_PROPS

echo "=== Step 5: Verify ISR is fully recovered ==="
kafka-topics.sh \
  --bootstrap-server $BOOTSTRAP \
  --describe \
  --under-replicated-partitions
# Should return no output (zero under-replicated partitions)

echo "=== Step 6: Check consumer lag has not grown ==="
kafka-consumer-groups.sh \
  --bootstrap-server $BOOTSTRAP \
  --describe \
  --group labs-api-group \
  --command-config $ADMIN_PROPS

4. Cluster failover runbook (DR event)

#!/bin/bash
# cluster-failover.sh — executed when primary is confirmed unrecoverable
# Decision gate: get explicit sign-off before executing

PRIMARY="primary-broker:9092"
BACKUP="backup-broker:9092"
ADMIN_PROPS="/tmp/backup-admin.properties"

echo "=== Step 1: Confirm primary is TRULY down (avoid split-brain) ==="
for attempt in 1 2 3; do
  kafka-topics.sh --bootstrap-server $PRIMARY --list && {
    echo "PRIMARY IS STILL REACHABLE — abort failover"
    exit 1
  }
  sleep 10
done
echo "Primary confirmed unreachable after 3 attempts — proceeding"

echo "=== Step 2: Check MM2 replication lag on backup ==="
kafka-consumer-groups.sh \
  --bootstrap-server $BACKUP \
  --describe \
  --group "mm2-primary->backup-labs.events-0" \
  --command-config $ADMIN_PROPS
# Note the LAG value — this is your data loss exposure (RPO gap)

echo "=== Step 3: Stop MM2 to prevent conflicts ==="
kubectl scale deployment mm2 --replicas=0 -n kafka

echo "=== Step 4: Reset consumer offsets on backup ==="
# Option A: to latest (accept data loss equal to MM2 lag)
kafka-consumer-groups.sh \
  --bootstrap-server $BACKUP \
  --group labs-api-group \
  --reset-offsets \
  --to-latest \
  --all-topics \
  --execute \
  --command-config $ADMIN_PROPS

# Option B: use MM2 offset translation (zero-loss, requires RemoteClusterUtils)
# See Day 54 for RemoteClusterUtils example

echo "=== Step 5: Redirect apps to backup cluster ==="
kubectl set env deployment/labs-api \
  KAFKA_BOOTSTRAP_SERVERS=$BACKUP \
  -n production

kubectl set env deployment/labs-socket \
  KAFKA_BOOTSTRAP_SERVERS=$BACKUP \
  -n production

kubectl rollout status deployment/labs-api -n production

echo "=== Step 6: Verify message flow on backup ==="
kafka-console-consumer.sh \
  --bootstrap-server $BACKUP \
  --topic primary.labs.events \
  --from-beginning \
  --max-messages 5

echo "=== Failover complete. Update incident runbook with timeline and RPO gap. ==="

Step 5 is missing a critical piece — see §7. It redirects bootstrap-servers but never touches the actual topic name the applications are configured to produce/consume — and Step 6’s own verification command (--topic primary.labs.events) reveals exactly why that’s a problem.

5. Backup strategy — what to back up

Export topic configs to Git (daily cron)

#!/bin/bash
# backup-topic-configs.sh
BOOTSTRAP="broker:9092"
BACKUP_DIR="/backup/kafka/$(date +%Y-%m-%d)"
mkdir -p $BACKUP_DIR

# Export all topic descriptions
kafka-topics.sh \
  --bootstrap-server $BOOTSTRAP \
  --describe \
  > $BACKUP_DIR/topics.txt

# Export consumer group offsets
kafka-consumer-groups.sh \
  --bootstrap-server $BOOTSTRAP \
  --list | while read group; do
    kafka-consumer-groups.sh \
      --bootstrap-server $BOOTSTRAP \
      --describe \
      --group $group \
      >> $BACKUP_DIR/consumer-groups.txt
  done

# Push to Git
cd /backup && git add . && git commit -m "Daily Kafka config backup $(date +%Y-%m-%d)" && git push

6. DR drill checklist — run quarterly

□ Simulate broker failure: kubectl delete pod labs-kafka-broker-1 -n kafka
  □ Verify leader election completes < 30s
  □ Verify under-replicated partitions return to 0 < 5 min
  □ Verify consumer lag did not grow during election

□ Simulate slow consumer: throttle labs-api processing
  □ Verify Grafana alert fires for lag > 1000
  □ Verify Slack notification received

□ Simulate full cluster failover (staging only):
  □ Execute cluster-failover.sh against staging clusters
  □ Measure actual RTO (time from declaring disaster to traffic flowing)
  □ Measure actual RPO (message count in MM2 lag at time of failover)
  □ Document gaps and update runbook

□ Verify backups are restorable:
  □ Restore topic configs from daily backup
  □ Confirm consumer group offsets can be reset correctly

7. Fixing the runbook — the missing topic-reconfiguration step

Step 6 of §4’s runbook verifies messages on primary.labs.events — which only exists because MM2’s default topic-prefixing policy was used (Day 54 §2/§3). But Step 5 never updates the applications’ own topic name configuration — it only redirects bootstrap-servers. If labs-api/labs-socket are still configured to produce/consume labs.events (their normal, day-to-day topic name), they will connect successfully to the backup cluster and then find nothing, because the replicated data actually landed under primary.labs.events, a topic name the application was never told about.

# What Step 5 in §4 is missing — updating the actual topic name, not just the broker address
kubectl set env deployment/labs-api \
  KAFKA_BOOTSTRAP_SERVERS=$BACKUP \
  KAFKA_TOPIC_NAME=primary.labs.events \
  -n production
# ...and labs-api's code needs to actually read KAFKA_TOPIC_NAME instead of a hardcoded "labs.events"

This is exactly why Day 54 §8 recommended IdentityReplicationPolicy for pure active-passive DR — with it, labs.events keeps its name on the backup cluster, Step 5’s bootstrap-server redirect is genuinely sufficient on its own, and Step 6’s verification command should read --topic labs.events, not primary.labs.events. Pick one and make the runbook and the MM2 config agree:

If using IdentityReplicationPolicy: Step 5 as originally written is correct and complete; fix Step 6’s verification command to drop the primary. prefix.

If keeping MM2’s default prefixing (e.g. because this cluster pair might later become active-active): Step 5 must be extended to also reconfigure the application’s topic name, not just its bootstrap servers — and that requires the application itself to support topic name as an externalized config value in the first place, which is a code-level requirement to verify ahead of any actual incident, not something to discover while executing the runbook under pressure.

Why this is worth catching now rather than during a drill: a DR runbook with an internal inconsistency like this can look complete and pass a superficial review, but fails at the exact moment it’s needed — this is precisely the kind of gap Day 54 §11’s “DR drill, not one-time verification” advice exists to catch, and precisely the kind of gap that’s cheaper to find by careful reading than by a failed real failover.

8. RPO has two different meanings — don’t conflate them

§1’s table lists a single RPO target (“near-zero with RF=3 + acks=all”) for both rows in §2’s failure scenario table — but “single broker crash” and “majority of brokers lost / full cluster failure” have RPO governed by completely different mechanisms, and treating them as the same number is misleading.

Why this distinction changes how you should read §4’s runbook: Step 2’s kafka-consumer-groups.sh --describe against MM2’s consumer group and its comment “this is your data loss exposure” is the actual RPO number for a cluster-level DR event — not the “near-zero” figure from §1’s table, which only applies to the single-broker scenario. A team that set an SLA expecting “near-zero RPO always” based on §1’s table alone will be surprised the first time a real cluster failover shows a non-trivial MM2 lag gap. State cluster-level DR RPO as “bounded by current MM2 replication lag, monitored continuously” rather than borrowing the single-broker RF/acks figure.

9. Manual sign-off vs automated failover — why this runbook is deliberately manual

This lesson’s own title promises “automated failover with health checks,” but §4’s actual runbook requires explicit human sign-off before executing (the comment directly above the script) and a manual three-attempt confirmation loop before proceeding. This is a deliberate, defensible design choice worth understanding rather than an oversight to “fix” by automating everything.

Why full automation is risky specifically for cluster-level failover:

False-positive risk: an automated system that decides “primary is down” based on a transient network blip (rather than genuine primary failure) and automatically triggers failover can cause exactly the split-brain scenario §4 Step 1 is designed to prevent — both clusters believing they’re primary, accepting writes independently, with no way to reconcile afterward.
Failover itself has cost — Step 4’s offset reset and Step 5’s application redirect are not free, reversible operations; triggering them automatically on a false alarm creates a second incident on top of (or instead of) the first.

What “automated failover with health checks” should actually mean in practice:

Automate: detection (health checks, alerting — Day 48) and the MECHANICAL steps
          within the runbook (offset translation, app config updates) once a human
          has confirmed the disaster is real

Keep manual: the actual GO/NO-GO decision to declare a disaster and trigger failover

The right target for automation is reducing RTO’s execution time, not removing the human decision. Health checks and alerting (Day 48) should make Step 1’s “confirm primary is truly down” fast and confident rather than a source of RTO delay; scripting Steps 2–6 (as this runbook already does) removes execution-time human error and speeds up the mechanical work. But the decision gate itself — “yes, this is a real disaster, execute the failover” — is exactly the kind of high-consequence, hard-to-reverse decision that benefits from a human in the loop, even in an otherwise highly automated operation.

10. Testing backup restoration, not just backup creation

§5’s backup strategy verifies data gets exported (backup-topic-configs.sh runs, commits succeed) — but an untested backup is only a belief that recovery will work, not a verified capability. §6’s checklist does include “verify backups are restorable” as a bullet, but it’s worth expanding on what that actually needs to mean.

# Restoration test — not just "the backup file exists" but "the backup file
# actually reconstructs a working configuration"
#!/bin/bash
# restore-drill.sh — run against a throwaway test cluster, not production

BACKUP_DIR="/backup/kafka/2026-08-20"   # a real historical backup, not a fresh one

# Recreate topics from the backed-up descriptions — this is the actual test:
# does the backup contain everything needed to reconstruct the cluster state,
# or does it silently assume some config lives elsewhere (ACLs? Schema Registry? Day 54 §9)
while read -r topic_line; do
  # parse topic name + partition count + config from topics.txt and recreate
  : # actual parsing/recreation logic
done < "$BACKUP_DIR/topics.txt"

# Attempt to reset a consumer group to the backed-up offsets and confirm
# it lands where expected — not just that the command runs without error

What a restoration test actually needs to prove: not “the export command ran successfully” (§5’s cron job already confirms that), but “given only this backup and a blank cluster, can the described state actually be reconstructed” — which surfaces gaps like missing ACL exports (§5’s table doesn’t explicitly cover ACLs, only “topic configs”), missing Schema Registry state (Day 54 §9’s gap, which applies to backup/restore just as much as to MM2 replication), or a Git backup that references secrets (TLS certs, credentials) stored somewhere the restoration process doesn’t actually have access to. Schedule this as its own periodic exercise, distinct from the DR failover drill in §6, since it tests a different failure mode (data/config loss requiring reconstruction from backup, vs cluster unavailability requiring failover to a live standby).

11. Common pitfalls

Redirecting bootstrap-servers in the failover runbook without also addressing topic naming — if MM2’s default prefixing is in use, applications silently connect to the right cluster but the wrong (nonexistent, from their perspective) topic name (§7)
Quoting a single RPO number for both single-broker and full-cluster-failure scenarios — these are governed by entirely different mechanisms (RF/acks vs MM2 lag) and conflating them sets the wrong SLA expectation (§8)
Treating “automated failover with health checks” as “remove the human decision entirely” — full automation of the GO/NO-GO decision risks triggering split-brain on a false positive; automate detection and mechanical execution, keep the decision gate manual (§9)
Verifying that backups are created without verifying they can be restored — an export job succeeding says nothing about whether the backup actually contains everything needed to reconstruct cluster state (§10)
Not testing the DR runbook’s internal consistency before a drill — a runbook that references primary.labs.events in one step and never updates application topic config in an earlier step is a documentation bug that a careful read-through can catch before it’s discovered mid-incident (§7)

Key Takeaways

Define RTO and RPO before writing any runbook — but recognize RPO means different things for single-broker failure (RF/acks-bounded, near-zero) vs full cluster DR (MM2-lag-bounded, monitor continuously)
Single broker failure is self-healing with RF=3 — monitor under-replicated partitions
Cluster failover: confirm down → check MM2 lag → reset offsets → redirect apps and reconcile topic naming (Day 54 §8) → verify
Never failover without confirming primary is truly down — split-brain causes duplicates, and this decision gate should stay human even as detection/execution are automated
Infrastructure config in Git is your most important backup — version everything, and periodically prove it can actually be restored, not just that it exports successfully
Automate detection and mechanical runbook steps to reduce RTO; keep the actual failover decision manual to avoid automating your way into split-brain
Practice the runbook quarterly — a DR drill reveals gaps before a real incident does, including internal inconsistencies (like §7’s) that a careful reading can also catch ahead of time

Support me through GitHub Sponsors.

Thank you for Reading !! See you in the next post.

➡️ Day 56: Prod cluster capstone — Spring Boot + Kafka 4 on K8s

Resources

📘 Kafka: The Definitive Guide — Chapters 8 & 10
🌐 kafka.apache.org/documentation/#basic_ops_restarting