60-Day Kafka 4 Learning Plan · Week 2— Day 8 of 60

60-Day Kafka 4 Learning Plan · Week 2 — Kafka 4 Internals Sources: Kafka: The Definitive Guide Ch.2 · kafka.apache.org/43/documentation/#kraft · KIP-500

Goal

Understand how KRaft replaced ZooKeeper in Kafka 4, how the @metadata topic works as the cluster’s source of truth, how Raft consensus achieves controller election, how snapshots bound recovery time, and how to monitor and operate a KRaft quorum in production.

1. Why KRaft replaced ZooKeeper

Kafka 4 is KRaft-only. ZooKeeper mode has been removed entirely.

2 . The @metadata topic

@metadata is an internal, single-partition, replicated Kafka topic that acts as the single source of truth for all cluster state. Every controller and broker continuously reads from it.

Record types written to @metadata

All these records are append-only — changes are expressed as new records, not overwrites.

3. Raft consensus — how controllers agree

KRaft uses the Raft distributed consensus protocol for the controller quorum.

Active Controller          @metadata log          Controller 2    Controller 3
   (leader, epoch N)  ──►  offset 0 … N    ──►   (follower)      (follower)
        write                replicated                ack             ack
                                │
                         majority ack → committed

Key rules:

One active controller (leader) accepts writes.
A write is committed only when a majority of the quorum acknowledges it (e.g. 2 of 3).
If the active controller fails → remaining nodes hold a Raft vote to elect a new leader.
Failover completes in milliseconds (no ZK session timeout to wait for).

Typical quorum sizes:

3 controllers → tolerate 1 failure
5 controllers → tolerate 2 failures

Why always odd numbers? A quorum of 4 tolerates the same 1 failure as a quorum of 3 (majority of 4 is 3, same as majority of 3 is 2 — you gain no extra fault tolerance but pay for an extra node). Always size controller quorums at 3 or 5, never an even number.

4. Leader epoch — preventing split-brain

Every controller election increments the epoch number. This epoch is embedded in every @metadata record.

Brokers reject any write from a controller with an old epoch — even if a previously crashed controller comes back online and tries to write stale records.

Epoch 5 (old controller, crashed)  →  write rejected by brokers
Epoch 6 (new controller, elected)  →  write accepted

Brokers track the high-water mark of the @metadata log. A broker that is behind must fetch all missing records before it can serve client requests.

5. Snapshots — bounded recovery time

Without snapshots, every broker restart must replay the entire @metadata log from offset 0 — this becomes prohibitively slow in large clusters with millions of partition records.

Solution: periodic snapshots checkpoint the full metadata state at a given offset.

[offset 0–5000] [5001–10k]  ← snapshot @10k →  [10001–15k] [15001–20k]
                                                 ↑ only this delta replayed on restart

Recovery = load snapshot + replay delta only. This bounds recovery time regardless of total log size.

Snapshot tuning (server.properties)

# Trigger snapshot after 20 MB of records written
metadata.log.max.record.bytes.between.snapshots=20971520

# Maximum idle time before forcing a snapshot
metadata.max.idle.interval.ms=500

Snapshots are stored in the metadata.log.dir directory alongside the log segments.

6. Broker recovery flow

1. Broker starts → contacts the active controller
2. Controller sends latest snapshot (if broker is far behind)
3. Broker loads snapshot → instantly knows full cluster state up to snapshot offset
4. Broker replays only delta records since the snapshot offset
5. Broker is caught up → registers with cluster, starts serving client requests

This 5-step process replaces the complex ZooKeeper session handshake and controller-initiated metadata push.

7. Key KRaft configuration reference

# server.properties (KRaft mode)

# Role of this node
process.roles=broker,controller       # combined node (dev)
# process.roles=broker               # broker-only (production)
# process.roles=controller           # controller-only (production)

# Unique node identifier (replaces broker.id)
node.id=1

# All controller nodes in the quorum
controller.quorum.voters=1@controller1:9093,2@controller2:9093,3@controller3:9093

# Separate directory for @metadata log
metadata.log.dir=/var/kafka/metadata

# Listener name used for controller-to-controller communication
controller.listener.names=CONTROLLER

# Listener config
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT

Initialise a new KRaft cluster

# Generate a unique cluster ID
CLUSTER_ID=$(kafka-storage.sh random-uuid)

# Format storage on every node
kafka-storage.sh format \
  --config /etc/kafka/server.properties \
  --cluster-id $CLUSTER_ID

8. Combined vs dedicated controller nodes

Why dedicate in production: a broker under heavy client load (GC pauses, disk I/O contention) can slow down controller responsiveness if they share a JVM, risking slower leader elections or metadata propagation exactly when the cluster is already stressed. Dedicated controllers keep metadata operations fast and predictable regardless of broker load.

Sizing rule: 3 dedicated controller nodes handle metadata for clusters up to very large broker counts — the @metadata topic scales with topic/partition churn, not with data throughput, so controller nodes can be modest (fewer cores, moderate RAM) compared to data brokers.

9. Observers — read scaling without voting rights

Beyond the voting quorum (controller.quorum.voters), Kafka 4 supports observers: brokers that replicate the @metadata log for local reads but don’t participate in Raft voting or count toward quorum majority.

# On a broker acting as an observer (not in controller.quorum.voters)
process.roles=broker
controller.quorum.bootstrap.servers=controller1:9093,controller2:9093,controller3:9093

Why this matters: every broker in the cluster needs fast, local access to metadata (topic configs, partition leadership) to serve Metadata API requests to clients efficiently. Observers get this without adding to the fault-tolerance math of the voting quorum — you can scale broker count freely without ever needing to resize (and risk) the controller quorum itself.

10. Monitoring KRaft health

# Inspect the live in-memory metadata tree on any node
kafka-metadata-shell.sh --snapshot /var/kafka/metadata/__cluster_metadata-0/00000000000000000000.checkpoint

# Dump and decode the raw @metadata log for debugging
kafka-dump-log.sh --cluster-metadata-decoder \
  --files /var/kafka/metadata/__cluster_metadata-0/00000000000000010000.log

Alerting priority: treat CurrentControllerId flapping (repeated re-elections) as a P1 signal — it usually means network partitioning or resource exhaustion on the controller nodes, and every flip briefly pauses cluster-wide metadata propagation.

11. Disaster recovery — quorum loss scenarios

Operational takeaway: losing quorum majority doesn’t stop existing produce/consume traffic on already-assigned partitions, but it freezes anything requiring a metadata change — no new topics, no reassignments, no config updates — until majority is restored.

Key Takeaways

KRaft eliminates ZooKeeper — @metadata topic IS the source of truth for all cluster state
Raft consensus: active controller writes, majority of quorum must acknowledge before commit — always size quorums at odd numbers (3 or 5)
Leader epoch prevents split-brain — brokers reject writes from controllers with stale epochs
Snapshots bound recovery time — only the delta since the last snapshot needs to be replayed
Controller failover takes milliseconds vs 30–60 s with ZooKeeper
Production clusters should use dedicated controller nodes, not combined broker+controller roles
Observers let brokers read metadata locally without affecting quorum fault tolerance
Losing quorum majority freezes metadata changes cluster-wide — existing traffic keeps flowing, but nothing new can be created or reassigned
Kafka 4 is KRaft-only — process.roles, node.id, controller.quorum.voters replace ZK config

Support me through GitHub Sponsors.

➡️ Day 9: Log segments — how Kafka 4 stores messages on disk

Resources

📘 Kafka: The Definitive Guide — Chapter 2 (KRaft: Kafka Without ZooKeeper)
🌐 kafka.apache.org/43/documentation/#kraft
🌐 KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum
🌐 KIP-595: Raft protocol for the metadata quorum

👉 Link to Medium blog