Kafka

60-Day Kafka 4 Learning Plan · Week 1 — Day 3 of 60

Editorial Team

60-Day Kafka 4 Learning Plan · Week 1 — Foundations
Sources: Kafka: The Definitive Guide Ch.6 · kafka.apache.org/43 — KRaft docs

Goal

Understand how Kafka 4 manages itself without ZooKeeper — and why that matters for every deployment decision.

1. Why KRaft? The ZooKeeper problem

Old architecture (Kafka ≤ 3.x)

Broker 1 (controller)  ─┐
Broker 2               ─┼──► ZooKeeper ensemble (3 separate nodes)
Broker 3               ─┘    stores: topics, partitions, ISR, ACLs, broker IDs

Problems with ZooKeeper:

  • 2 systems to deploy, monitor, and upgrade
  • Metadata lag between ZooKeeper and Kafka brokers
  • Hard ceiling of ~200k partitions per cluster
  • Slow controller failover (seconds, not milliseconds)
  • Complex rolling upgrades across two systems

New architecture (Kafka 4 — KRaft only)

┌────────────────────────────────────────────────────────────────┐
│  One unified Kafka cluster — metadata managed internally       │
│                                                                │
│  [Controller 1]   [Controller 2]   [Controller 3]   [Brokers]  │
│  ACTIVE LEADER    hot standby      hot standby       N nodes   │
└────────────────────────────────────────────────────────────────┘

KRaft benefits:

  • ✓ One system — simpler ops, no ZooKeeper to monitor or fail
  • ✓ Metadata always inside Kafka — no lag, strongly consistent
  • ✓ Millions of partitions per cluster (vs ~200k with ZooKeeper)
  • ✓ Controller failover in milliseconds vs seconds

2. How Raft consensus works

Every controller node is always in one of 3 states:

Leader election — 4 steps

1. Heartbeat        2. Timeout          3. Vote request     4. Majority wins
┌──────────┐       ┌──────────┐        ┌──────────┐        ┌──────────┐
│ Leader   │ ───►  │ No hbeat │ ───►   │ Candidate│ ───►   │ New      │
│ pings    │       │ follower │        │ asks     │        │ leader   │
│ followers│       │→candidate│        │ peers    │        │ elected  │
└──────────┘       └──────────┘        └──────────┘        └──────────┘

Quorum rules:

  • 3 controllers → need 2 votes to elect leader
  • 5 controllers → need 3 votes to elect leader
  • Elections use term numbers — higher term always wins, prevents split-brain scenarios

3. The __cluster_metadata log

Every cluster change is appended as a record to this special internal topic:

offset 0      offset 1      offset 2      offset 3      offset 4      offset 5      offset 6 →
┌───────────┬─────────────┬─────────────┬────────────┬──────────────┬───────────┬────────────┐
│ BrokerReg │ TopicCreate │ PartAssign  │ ISRChange  │ ConfigChange │ ACLUpdate │ next...    │
└───────────┴─────────────┴─────────────┴────────────┴──────────────┴───────────┴────────────┘

What gets stored:

Snapshots: Periodically the full metadata state is compacted into a snapshot file on disk. New controllers can bootstrap from the latest snapshot without replaying the entire log from offset 0.

4. KRaft process roles and deployment modes

Set process.roles in server.properties:

Production recommendation:

  • 3 dedicated controller nodes + N broker nodes
  • Always use an odd number of controllers (3 or 5)
  • Controller nodes need: 5 GB RAM + 5 GB disk for the metadata log
  • Do not use broker,controller combined mode for large production workloads

5. How metadata flows — full picture

Client              Active Controller     Follower Ctrls     Brokers
  │                        │                    │               │
  │── 1. create topic ────►│                    │               │
  │                        │── 2. replicate ───►│               │
  │                        │◄─────── quorum ack─┤               │
  │                        │── 3. committed ────┤               │
  │                        │                    │               │
  │                        │◄──────────── 4. brokers fetch metadata ◄─┤
  │                        │                                    │
  │                        │──────────── 5. metadata applied ──►│
  │                                           topic now live    │

6. Useful KRaft commands

# Check quorum status
kafka-metadata-quorum.sh --bootstrap-controller :9093 describe --status

# Describe metadata log replication
kafka-metadata-quorum.sh --bootstrap-controller :9093 describe --replication

# Browse metadata shell (interactive)
kafka-metadata-shell.sh --directory /var/kafka-logs/__cluster_metadata-0

# Generate cluster ID (once, before formatting storage)
kafka-storage.sh random-uuid

# Format storage on each node (run once per node)
kafka-storage.sh format \
  --config /etc/kafka/server.properties \
  --cluster-id <YOUR_CLUSTER_ID>

TL;DR

KRaft replaces ZooKeeper by running the Raft consensus algorithm inside Kafka itself. A quorum of controller nodes (3 recommended) elect one active leader that handles all metadata writes. Every cluster change — topic creation, partition assignment, ISR update, ACL change — is appended to __cluster_metadata, replicated to follower controllers for quorum acknowledgment, then fetched by brokers. Snapshots enable fast bootstrap. Kafka 4 is KRaft-only: ZooKeeper is permanently removed.

Support me through GitHub Sponsors.

Next

➡️ Day 4 — Local setup: Docker Compose + Spring Boot 3.x + Kafka 4 (KRaft mode)

Resources

👉 Link to Medium blog

,

Related Posts

Kafka

60-Day Kafka 4 Learning Plan · Week 3 — Day 16 of 60

Editorial Team
Kafka

60-Day Kafka 4 Learning Plan · Week 3 — Day 15 of 60

Editorial Team