60-Day Kafka 4 Learning Plan · Week 2 — Day 14 of 60

60-Day Kafka 4 Learning Plan · Week 2 — Kafka 4 Internals (final day)
Sources: Kafka: The Definitive Guide Ch.6 · kafka.apache.org/43/documentation/#operations

Goal

Hands-on lab tying together everything from Week 2: inspect raw @metadata log segments, check ISR and replication state, simulate a broker failure, and tune durability settings.

Prerequisites

• Docker & Docker Compose
• 4 GB RAM available to Docker
• Basic Kafka familiarity (topics, producers, consumers)
• curl or any HTTP client (optional, for Kafka UI)

KRaft Architecture — What Changed in Kafka 4

Before writing a single command, it’s worth understanding what KRaft replaced and why.

ZooKeeper era: Kafka stored cluster metadata (broker registrations, partition assignments, leader epochs) in ZooKeeper — a separate consensus system that Kafka operators had to run, monitor, and scale independently. ZooKeeper and Kafka had different availability models, different operational tooling, and different failure modes.

KRaft era (Kafka 4 — ZooKeeper fully removed): Kafka uses an internal Raft consensus protocol to manage its own metadata. The metadata is stored in a special internal partition called __cluster_metadata (also referred to as the @metadata log). Every broker registration, partition assignment, ISR change, and leader election is appended to this log as a durable, replicated record batch.

In combined mode — what this lab runs — each Kafka node acts as both a broker (serves producers and consumers) and a controller (participates in Raft consensus for metadata management). This eliminates the broker-vs-controller split and makes single-node and small clusters significantly simpler to operate.

Cluster Architecture

┌──────────────────────────────────────────────────────────────┐
│                  Docker Compose Network                      │
│                                                              │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐        │
│  │   kafka-1    │  │   kafka-2    │  │   kafka-3    │        │
│  │ NodeId: 1    │  │ NodeId: 2    │  │ NodeId: 3    │        │
│  │ Roles:       │  │ Roles:       │  │ Roles:       │        │
│  │   broker     │  │   broker     │  │   broker     │        │
│  │   controller │  │   controller │  │   controller │        │
│  │              │  │              │  │              │        │
│  │ CONTROLLER   │  │ CONTROLLER   │  │ CONTROLLER   │        │
│  │ :9093 ◄──────┼──┼──────────────┼──┼── Raft       │        │
│  │ INTERNAL     │  │ INTERNAL     │  │ INTERNAL     │        │
│  │ :29092 ◄─────┼──┼── replication┼──┼──────────►   │        │
│  │ EXTERNAL     │  │ EXTERNAL     │  │ EXTERNAL     │        │
│  │ :9092        │  │ :9092        │  │ :9092        │        │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘        │
│         │                 │                  │               │
│  ┌──────▼─────────────────▼──────────────────▼───────┐       │
│  │                    kafka-ui :8080                 │       │
│  └───────────────────────────────────────────────────┘       │
└──────────────────────────────────────────────────────────────┘
      │                  │                   │
localhost:9092     localhost:9093       localhost:9094
  (kafka-1)          (kafka-2)           (kafka-3)
                                          Kafka UI → localhost:8090

Three listeners per node:

The KAFKA_ADVERTISED_LISTENERS for INTERNAL uses kafka-N:29092 (resolvable inside Docker). For EXTERNAL it uses localhost:9092/9093/9094 (resolvable from the host). Kafka UI connects to all three nodes via the INTERNAL listener.

services:

  # ── Broker / Controller 1 ────────────────────────────────────────────────
  kafka-1:
    image: apache/kafka:4.0.0
    hostname: kafka-1
    container_name: kafka-1
    ports:
      - "9092:9092"   # EXTERNAL — host connects here
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      CLUSTER_ID: 4L6g3nShT-eMCtK--X86sw

      # All three nodes vote in the Raft quorum
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093

      # Three listeners per node:
      #   CONTROLLER  — Raft consensus (internal, never advertised externally)
      #   INTERNAL    — inter-broker replication (Docker network)
      #   EXTERNAL    — client connections from the host machine
      KAFKA_LISTENERS: CONTROLLER://kafka-1:9093,INTERNAL://kafka-1:29092,EXTERNAL://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-1:29092,EXTERNAL://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER

      # Cluster-wide replication defaults (3 nodes → RF=3 is possible)
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2
      KAFKA_MIN_INSYNC_REPLICAS: 2
      KAFKA_DEFAULT_REPLICATION_FACTOR: 3
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0

      KAFKA_LOG_DIRS: /var/lib/kafka/data
    volumes:
      - kafka-1-data:/var/lib/kafka/data
    healthcheck:
      test: ["CMD-SHELL", "/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 > /dev/null 2>&1"]
      interval: 15s
      timeout: 10s
      retries: 10
      start_period: 60s

  # ── Broker / Controller 2 ────────────────────────────────────────────────
  kafka-2:
    image: apache/kafka:4.0.0
    hostname: kafka-2
    container_name: kafka-2
    ports:
      - "9093:9092"   # EXTERNAL — host connects here (note: maps container:9092)
    environment:
      KAFKA_NODE_ID: 2
      KAFKA_PROCESS_ROLES: broker,controller
      CLUSTER_ID: 4L6g3nShT-eMCtK--X86sw
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093
      KAFKA_LISTENERS: CONTROLLER://kafka-2:9093,INTERNAL://kafka-2:29092,EXTERNAL://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-2:29092,EXTERNAL://localhost:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2
      KAFKA_MIN_INSYNC_REPLICAS: 2
      KAFKA_DEFAULT_REPLICATION_FACTOR: 3
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_LOG_DIRS: /var/lib/kafka/data
    volumes:
      - kafka-2-data:/var/lib/kafka/data
    healthcheck:
      test: ["CMD-SHELL", "/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 > /dev/null 2>&1"]
      interval: 15s
      timeout: 10s
      retries: 10
      start_period: 60s

  # ── Broker / Controller 3 ────────────────────────────────────────────────
  kafka-3:
    image: apache/kafka:4.0.0
    hostname: kafka-3
    container_name: kafka-3
    ports:
      - "9094:9092"   # EXTERNAL — host connects here
    environment:
      KAFKA_NODE_ID: 3
      KAFKA_PROCESS_ROLES: broker,controller
      CLUSTER_ID: 4L6g3nShT-eMCtK--X86sw
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093
      KAFKA_LISTENERS: CONTROLLER://kafka-3:9093,INTERNAL://kafka-3:29092,EXTERNAL://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-3:29092,EXTERNAL://localhost:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2
      KAFKA_MIN_INSYNC_REPLICAS: 2
      KAFKA_DEFAULT_REPLICATION_FACTOR: 3
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_LOG_DIRS: /var/lib/kafka/data
    volumes:
      - kafka-3-data:/var/lib/kafka/data
    healthcheck:
      test: ["CMD-SHELL", "/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 > /dev/null 2>&1"]
      interval: 15s
      timeout: 10s
      retries: 10
      start_period: 60s

  # ── Kafka UI ─────────────────────────────────────────────────────────────
  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    ports:
      - "8090:8080"
    depends_on:
      kafka-1:
        condition: service_healthy
      kafka-2:
        condition: service_healthy
      kafka-3:
        condition: service_healthy
    environment:
      KAFKA_CLUSTERS_0_NAME: kraft-cluster
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka-1:29092,kafka-2:29092,kafka-3:29092

volumes:
  kafka-1-data:
  kafka-2-data:
  kafka-3-data:

Lab 0 — Start the Cluster

docker compose up -d

Wait for all three nodes to pass their healthchecks (~60 seconds on first pull):

for container in kafka-1 kafka-2 kafka-3; do
  until [ "$(docker inspect --format='{{.State.Health.Status}}' "$container")" = "healthy" ]; do
    printf "."
    sleep 3
  done
  echo " $container healthy"
done

Verify KRaft Quorum

Kafka 4 CLI change — --bootstrap-controller required In Kafka 3.x, kafka-metadata-quorum.sh accepted --bootstrap-server (broker address). In Kafka 4, it connects directly to the KRaft controller quorum via the CONTROLLER listener and requires --bootstrap-controller <host:controllerPort> instead. All other admin tools (kafka-topics.sh, kafka-configs.sh, kafka-log-dirs.sh, etc.) still use --bootstrap-server — only kafka-metadata-quorum.sh changed.

docker exec kafka-1 /opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-controller kafka-1:9093 \
  describe --status

Expected output:

Reading the output field by field:

• LeaderId: 3 — kafka-3 won the initial Raft leader election and is the active controller. Any node can win; this is normal.
• LeaderEpoch: 1 — the leader has never changed since cluster start. Each leader change increments this monotonically.
• HighWatermark: 1378 — 1 378 metadata records have been committed to the @metadata log. A freshly started 3-node cluster generates hundreds of records during broker registration, feature negotiation, and internal topic creation. The value grows with every topology change.
• MaxFollowerLag: 0 — all followers are fully caught up; no node is behind on metadata replication.
• MaxFollowerLagTimeMs: 224 — the largest gap between a follower’s last fetch and now, in milliseconds. Sub-second lag on a healthy cluster is expected; high values (seconds or more) indicate a slow or overloaded follower.
• CurrentVoters — Kafka 4 changed this field from a plain integer list ([1, 2, 3] in Kafka 3.x) to a JSON array of voter descriptors, each carrying the node id, a directoryId (used with JBOD storage; null for single-volume brokers), and the CONTROLLER listener endpoints. All three nodes are listed — the quorum is complete.
• CurrentObservers: [] — no observer nodes (read-only Raft members that replicate metadata but do not vote).

LeaderId tells you which node is the active controller — the one currently making authoritative metadata decisions. This changes on leader crash or deliberate re-election.

Check replication across the quorum:

docker exec kafka-1 /opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-controller kafka-1:9093 \
  describe --replication

All three nodes at the same LogEndOffset (1378) with Lag: 0. The leader (kafka-3) and both followers are fully in sync — the cluster is ready.

Open Kafka UI: http://localhost:8090 → kraft-cluster. You’ll see 3 brokers in the Brokers tab.

Lab 1 — Inspect the @metadata Log

The __cluster_metadata-0 partition is the Raft log. Every change to cluster state — broker registrations, partition assignments, ISR updates, leader elections — is written here as a durable record batch before taking effect.

List the log segments

docker exec kafka-1 ls -lh /var/lib/kafka/data/__cluster_metadata-0/

The naming convention matches regular Kafka log segments: the filename is the base offset of the segment.

Dump and read the metadata log

docker exec kafka-1 /opt/kafka/bin/kafka-dump-log.sh \
  --files /var/lib/kafka/data/__cluster_metadata-0/00000000000000000000.log \
  --print-data-log 2>/dev/null | head -80

You’ll see record batches like this:

Record types you’ll see:

Count record types

META_LOG_DIR="/var/lib/kafka/data/__cluster_metadata-0"
LOG_FILE=$(docker exec kafka-1 sh -c "ls $META_LOG_DIR/*.log | sort | tail -1")

docker exec kafka-1 /opt/kafka/bin/kafka-dump-log.sh \
  --files "$LOG_FILE" \
  --cluster-metadata-decoder \
  --print-data-log 2>/dev/null \
  | grep '"type"' | grep -oP '"type"\s*:\s*"\K[^"]+' | sort | uniq -c | sort -rn

3 REGISTER_CONTROLLER_RECORD
1 FEATURE_LEVEL_RECORD
1 PRODUCER_IDS_RECORD

This is what a fresh 3-node cluster looks like before any topics are created: three broker registrations and a feature level record.

Lab 2 — Create Topic and Produce Messages

Create the orders topic with replication-factor=3 and min.insync.replicas=2:

docker exec kafka-1 /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka-1:29092 \
  --create \
  --topic orders \
  --partitions 3 \
  --replication-factor 3 \
  --config min.insync.replicas=2

Describe the topic immediately to see the initial partition-to-broker assignment:

docker exec kafka-1 /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka-1:29092 \
  --describe \
  --topic orders

Expected warning — safe to ignore You may see a transient WARN line before the topic output:

WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-1/...:29092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)

node -1 is Kafka’s internal label for the ephemeral bootstrap node — the initial connection the AdminClient makes before it has cluster metadata. This failure is a single failed attempt during the brief window between topic creation and leader assignment stabilizing. The client retries immediately and succeeds. If the describe output appears, the cluster is healthy

• Replicas: all nodes that hold a copy (regardless of whether they’re current)
• Isr: replicas that are in-sync — within replica.lag.time.max.ms (default 30s) of the leader

With a healthy 3-node cluster, Isr = Replicas for all partitions.

Check the @metadata log again — you’ll now see PartitionRecord entries for each of the 3 partitions:

docker exec kafka-1 /opt/kafka/bin/kafka-dump-log.sh \
  --files /var/lib/kafka/data/__cluster_metadata-0/00000000000000000000.log \
  --print-data-log 2>/dev/null | grep "type:" | sort | uniq -c | sort -rn

Produce 30 messages:

for i in $(seq 1 30); do
  echo "order-${i}:value-${i}" | \
    docker exec -i kafka-1 /opt/kafka/bin/kafka-console-producer.sh \
      --bootstrap-server kafka-1:29092 \
      --topic orders \
      --property "parse.key=true" \
      --property "key.separator=:"
done

Lab 3 — Diagnose ISR State and Replication Lag

Check ISR

docker exec kafka-1 /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka-1:29092 \
  --describe \
  --topic orders

All partitions should show Isr: 1,2,3 (full replication).

Check under-replicated partitions

docker exec kafka-1 /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka-1:29092 \
  --describe \
  --under-replicated-partitions

No output means no partitions are under-replicated. This is the green state.

Check per-replica log offsets and lag

docker exec kafka-1 /opt/kafka/bin/kafka-log-dirs.sh \
  --bootstrap-server kafka-1:29092 \
  --topic-list orders \
  --describe

The JSON output includes offsetLag per partition per broker. offsetLag: 0 on all follower replicas means they’re fully caught up to the leader.

Check under-min-ISR partitions (a stricter check)

docker exec kafka-1 /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka-1:29092 \
  --describe \
  --under-min-isr-partitions

No output = ISR size ≥ min.insync.replicas for all partitions. The cluster can accept acks=all writes safely.

Lab 4 — Kill kafka-3 (the KRaft Leader), Watch Re-election

This is the key lab. kafka-3 is the active KRaft controller (established in Lab 0). Killing it forces two simultaneous re-elections: a Raft leader election among the surviving controllers, and a partition leader election for every partition whose leader was on kafka-3.

Capture pre-kill state

docker exec kafka-1 /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka-1:29092 \
  --describe --topic orders

Note which partitions have Leader: 3 — those are the ones that will elect a new leader.

Stop kafka-3

docker compose stop kafka-3

Wait 5–10 seconds for the controller to detect the failure, then query via kafka-1:

docker exec kafka-1 /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka-1:29092 \
  --describe --topic orders

What happened:

• Leader: 3 on Partition 2 → elected a new leader from the remaining ISR members (1 or 2)
• Isr: 1,2,3 → shrunk to Isr: 1,2 on all partitions (kafka-3 removed from every ISR)
• Replicas: ...,3,... → unchanged (kafka-3 still listed; it’s offline but not deregistered)

Verify the KRaft quorum still has quorum

docker exec kafka-2 /opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-server kafka-2:29092 \
  describe --status

LeaderId: 1 — kafka-1 won the Raft election after kafka-3 went offline. LeaderEpoch: 2 confirms a new election ran (epoch was 1 when kafka-3 was leader; it increments monotonically each time a new leader is elected). The new controller kafka-1 then wrote the PartitionChangeRecord entries that removed kafka-3 from all ISRs and elected the new partition leaders.

CurrentVoters still lists kafka-3 as a registered voter even though it’s offline — Raft voter membership is a cluster configuration, not a liveness state. The quorum now commits with kafka-1 and kafka-2 (2 of 3 votes, which meets the majority threshold ⌊3/2⌋ + 1 = 2).

If you stopped 2 of 3 nodes instead, the surviving node would hold only 1 of 3 votes — below majority — and the cluster would become unavailable for metadata changes. Existing consumers could still read from known leaders, but anything requiring metadata (new connections, topic creation, leader queries) would fail.

Verify produces still work (ISR = 2 ≥ min.insync.replicas = 2)

echo "after-failure:value" | \
  docker exec -i kafka-1 /opt/kafka/bin/kafka-console-producer.sh \
    --bootstrap-server kafka-1:29092 \
    --topic orders \
    --property "parse.key=true" \
    --property "key.separator=:"

Produce succeeds. With min.insync.replicas=2 and ISR=[1,2], the durability contract is still met.

Check under-replicated partitions

docker exec kafka-1 /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka-1:29092 \
  --describe \
  --under-replicated-partitions

 Topic: orders Partition: 0 Leader: 1 Replicas: 3,1,2 Isr: 1,2 Elr:  LastKnownElr: 
 Topic: orders Partition: 1 Leader: 1 Replicas: 1,2,3 Isr: 1,2 Elr:  LastKnownElr: 
 Topic: orders Partition: 2 Leader: 2 Replicas: 2,3,1 Isr: 2,1 Elr:  LastKnownElr:

All three partitions are under-replicated (replica count 3, ISR size 2). This is the expected warning state — the cluster is operational but no longer fully redundant. Losing either kafka-1 or kafka-2 now would drop the ISR to 1, which is below min.insync.replicas=2, and produces would fail.

Lab 5 — Tune min.insync.replicas

This is the central operational decision for Kafka durability. min.insync.replicas (often abbreviated min.isr) defines how many replicas must acknowledge a write before it’s considered durable when acks=all.

kafka-3 is still stopped. ISR for all partitions is [1, 2] — size 2.

Scenario A — min.insync.replicas = 2 (current, ISR size = 2)

docker exec kafka-2 /opt/kafka/bin/kafka-configs.sh \
  --bootstrap-server kafka-2:29092 \
  --alter --entity-type topics --entity-name orders \
  --add-config min.insync.replicas=2

Produce with acks=all:

echo "scenario-a:test" | \
  docker exec -i kafka-2 /opt/kafka/bin/kafka-console-producer.sh \
    --bootstrap-server kafka-2:29092 \
    --topic orders --request-required-acks -1 \
    --property "parse.key=true" --property "key.separator=:"

Result: SUCCESS. ISR size (2) equals min.insync.replicas (2). The broker acknowledges the write.

Scenario B — min.insync.replicas = 3 (stricter, ISR size = 2)

docker exec kafka-2 /opt/kafka/bin/kafka-configs.sh \
  --bootstrap-server kafka-2:29092 \
  --alter --entity-type topics --entity-name orders \
  --add-config min.insync.replicas=3

Produce:

echo "scenario-b:test" | \
  docker exec -i kafka-2 /opt/kafka/bin/kafka-console-producer.sh \
    --bootstrap-server kafka-2:29092 \
    --topic orders --request-required-acks -1 \
    --property "parse.key=true" --property "key.separator=:"

Result: FAILURE. Error: org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required.

ISR size (2) is below min.insync.replicas (3). The broker refuses the write rather than risk acknowledging data that isn’t durably replicated. This is the durability guarantee: you will not lose data as long as you don’t lose all brokers in the ISR simultaneously.

Scenario C — min.insync.replicas = 1 (permissive)

docker exec kafka-2 /opt/kafka/bin/kafka-configs.sh \
  --bootstrap-server kafka-2:29092 \
  --alter --entity-type topics --entity-name orders \
  --add-config min.insync.replicas=1

Produce:

echo "scenario-c:test" | \
  docker exec -i kafka-2 /opt/kafka/bin/kafka-console-producer.sh \
    --bootstrap-server kafka-2:29092 \
    --topic orders --request-required-acks -1 \
    --property "parse.key=true" --property "key.separator=:"

Result: SUCCESS. But now if the partition leader crashes before the other replica replicates the write, that data is gone. min.insync.replicas=1 with acks=all is functionally equivalent to acks=1 — only the leader must write before acknowledging.

The tradeoff table

Production recommendation for RF=3: min.insync.replicas=2. This tolerates a single broker failure without losing durability or availability.

Restore the correct value:

docker exec kafka-2 /opt/kafka/bin/kafka-configs.sh \
  --bootstrap-server kafka-2:29092 \
  --alter --entity-type topics --entity-name orders \
  --add-config min.insync.replicas=2

Lab 6 — Recover kafka-3

docker compose start kafka-3

Wait for the healthcheck, then verify ISR restoration:

docker exec kafka-1 /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka-1:29092 \
  --describe --topic orders

Within 30 seconds you should see:

ISR is [1,2,3] again. kafka-3 reconnected, fetched the missed segments from each partition leader, and was re-admitted to the ISR once it caught up within replica.lag.time.max.ms.

Two things did not auto-rebalance:

1. Partition leadership — kafka-3 is the preferred replica for Partition 0 (first entry in Replicas: 3,1,2) but didn’t reclaim the leader role. Partition leaders don’t auto-migrate back on recovery to avoid unnecessary unavailability windows.
2. KRaft quorum leadership — kafka-3 was the Raft leader before the failure but rejoins as a follower. The new Raft leader (kafka-1) retains leadership.

Run a preferred replica election to restore the pre-failure partition leader balance:

docker exec kafka-1 /opt/kafka/bin/kafka-leader-election.sh \
  --bootstrap-server kafka-1:29092 \
  --election-type PREFERRED \
  --all-topic-partitions

After that, Partition 0 leadership moves back to kafka-3:

The KRaft quorum leader (kafka-1) will not automatically revert. That’s intentional — Raft doesn’t do preferred leader elections. If you need kafka-3 to reclaim the controller role, restart kafka-1 to trigger a new Raft election (not recommended during production without careful planning).

Inspect the @metadata log one final time to see the full event timeline:

META_LOG_DIR="/var/lib/kafka/data/__cluster_metadata-0"
LOG_FILE=$(docker exec kafka-1 sh -c "ls $META_LOG_DIR/*.log | sort | tail -1")

docker exec kafka-1 /opt/kafka/bin/kafka-dump-log.sh \
  --files "$LOG_FILE" \
  --cluster-metadata-decoder \
  --print-data-log 2>/dev/null \
  | grep '"type"' | grep -oP '"type"\s*:\s*"\K[^"]+' | sort | uniq -c | sort -rn

  31295 NO_OP_RECORD
      7 PARTITION_CHANGE_RECORD        ← ISR changes + leader elections
      7 BROKER_REGISTRATION_CHANGE_RECORD
      4 REGISTER_CONTROLLER_RECORD
      4 REGISTER_BROKER_RECORD
      3 PRODUCER_IDS_RECORD
      3 PARTITION_RECORD
      3 FEATURE_LEVEL_RECORD
      1 TOPIC_RECORD
      1 END_TRANSACTION_RECORD
      1 CONFIG_RECORD
      1 BEGIN_TRANSACTION_RECORD

Every state transition is durably recorded. This is what makes KRaft auditable in a way ZooKeeper never was: you can replay the cluster’s entire history from the @metadata log.

Performance Considerations

Replica fetch lag budget

replica.lag.time.max.ms (default: 30,000 ms) controls how long a follower can lag the leader before being removed from the ISR. Reducing this makes the ISR tighter — under-replication surfaces faster — but increases the risk of ISR thrashing on transient network slowness. Don’t reduce below 10,000 ms in production.

Metadata log compaction

The @metadata log is periodically compacted by the active controller into a snapshot. After a snapshot is written, log segments before the snapshot offset can be deleted. Tune metadata.log.max.record.bytes.between.snapshots to control snapshot frequency. More frequent snapshots mean faster broker restarts (less log replay) but more CPU overhead.

Controller quorum and latency

The Raft leader must commit metadata changes to a majority of voters before returning. With 3 nodes in a single data center, this adds ~1–3 ms to leader-election and topic-creation operations. Across data centers the commit latency mirrors your network RTT. Put controller voters in proximity when possible.

Preferred leader election

After rolling restarts or broker recovery, partition leaders drift away from their preferred replicas (the first entry in the Replicas list). Periodically run kafka-leader-election.sh --election-type PREFERRED to rebalance leadership and even out broker load.

Common Pitfalls

Same CLUSTER_ID across all nodes is mandatory. If each node generates a random CLUSTER_ID, they form independent clusters that refuse to communicate. Set a fixed CLUSTER_ID in docker-compose.yml (generate with /opt/kafka/bin/kafka-storage.sh random-uuid).

All three nodes must start for the quorum to form. Raft requires a majority to elect a leader. If only 1 of 3 nodes starts, it can’t elect a leader and the cluster hangs. With Docker Compose, start all nodes simultaneously — don’t try to start them one by one expecting the first to become leader.

KAFKA_CONTROLLER_QUORUM_VOTERS must list all nodes. Missing a node means it’s never eligible for leadership. Quorum arithmetic still works (the listed nodes can form a majority), but if the missing node later joins, it joins as a voter not in the official voter list, which requires a quorum reconfiguration.

Stopping 2 of 3 nodes makes the cluster metadata-unavailable. You can still consume from the surviving broker if you know partition offsets, but anything requiring metadata (new connections, leader queries, topic creation) will fail. This is correct Raft behavior — no split-brain.

Under-replicated ≠ down. A partition showing under-replicated-partitions is still available for reads and writes — it just doesn’t meet its replication target. Don’t conflate under-replication with unavailability.

min.insync.replicas has no effect when acks != all. With acks=1, the producer doesn’t wait for followers — min.insync.replicas is irrelevant. Both must be set for the durability guarantee to apply.

Key Takeaways

• The @metadata log is the source of truth for all cluster state in KRaft — kafka-dump-log.sh reads it directly from disk and reveals every topology change as a typed record
• A 3-node cluster tolerates 1 broker failure with no data loss and no availability interruption when min.insync.replicas=2 — losing 2 of 3 nodes makes the cluster metadata-unavailable, not data-corrupt
• min.insync.replicas is a contract with acks=all: it tells the leader to refuse a write rather than acknowledge data that isn’t safely replicated — tuning it is the primary lever between availability and durability

Week 2 recap

The complete lab scripts and Docker Compose file are available on GitHub.

Support me through GitHub Sponsors.

Next

➡️ Week 3 · Day 15: Spring Kafka 3.x — Boot autoconfiguration for Kafka 4

References

• Apache Kafka KRaft Mode Documentation
• KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum
• kafka-dump-log.sh Tool Reference
• kafka-metadata-quorum.sh Tool Reference
• Apache Kafka Docker Image
• Kafka Replication Design

👉 Link to Medium blog