60-Day Kafka 4 Learning Plan · Week 1 — Foundations
Sources: Kafka: The Definitive Guide Ch.1–2 · kafka.apache.org/43 — Design
Goal
Master the 6 core building blocks of Kafka — the vocabulary for every future day.
1. Message anatomy
A message is the unit of data in Kafka. It consists of:
Key rule: Same key → always same partition (via hash(key) mod numPartitions).
Batching: Messages are grouped into batches before sending — bigger batch = higher throughput, higher latency.
2. Topics
A topic is a named, durable log to which producers write and consumers subscribe.
- Identified by name:
orders,payments,user-events - Split into N partitions for parallelism
- Configured with a replication factor R for fault tolerance
- Retention configurable per topic:
- By time: 7 days (default)
- By size: 1 GB
- Log compaction: keep only the latest value per key — ideal for change-data-capture (CDC)
3. Partitions & Offsets
A partition is an ordered, append-only sub-log within a topic.
topic: "orders" — 3 partitions
Partition 0 (Broker 1 LEADER) Partition 1 (Broker 2 LEADER) Partition 2 (Broker 3 LEADER)
┌────┬────┬────┬────┐ ┌────┬────┬────┐ ┌────┬────┐
│ 0 │ 1 │ 2 │ 3→ │ │ 0 │ 1 │ 2→ │ │ 0 │ 1→ │
│ #1 │ #3 │ #5 │ │ │ #2 │ #4 │ │ │ #6 │ │
└────┴────┴────┴────┘ └────┴────┴────┘ └────┴────┘
append-only log append-only log append-only logKey rules:
- Ordering is guaranteed within a partition — not across partitions
- Offsets restart at 0 for each partition independently — offsets are partition-scoped
- Each partition lives on exactly one broker (the leader)
4. Consumer Offsets & Lag
Partition 0 log:
┌──┬──┬──┬──┬──┬──┬──┐
│ 0│ 1│ 2│ 3│ 4│ 5│ 6│ ← LOG END
└──┴──┴──┴──┴──┴──┴──┘
↑
committed offset = 3
└──────────┘
LAG = 2 (msgs 4 & 5 unread)
Consumers commit offsets to Kafka’s internal __consumer_offsets topic — this is how they resume after a restart without losing their place.
5. Brokers & Clusters
A broker is a single Kafka server process.
Broker 1 Broker 2 Broker 3
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ P0 (LEADER) │ │ P0 (FOLLOWER)│ │ P0 (FOLLOWER)│
│ P1 (FOLLOWER)│ │ P1 (LEADER) │ │ P1 (FOLLOWER)│
│ P2 (FOLLOWER)│ │ P2 (FOLLOWER)│ │ P2 (LEADER) │
│ KRaft Ctrl │ │ │ │ │
└──────────────┘ └──────────────┘ └──────────────┘- One broker handles thousands of partitions and millions of messages/sec
- Each partition has exactly one leader broker — all reads and writes go to the leader
- Kafka 4: KRaft controller (elected from broker quorum) replaces ZooKeeper entirely
- Cluster metadata lives in the internal
__cluster_metadatatopic — no external dependency
6. Replication & ISR
ISR = In-Sync Replica — a follower that has fully caught up with the leader’s log.
Partition 0 — replication factor = 3
[LEADER] replicate → [FOLLOWER ISR] replicate → [FOLLOWER ISR]
Broker 1 Broker 2 Broker 3
offset 0-5 synced replica synced replica
reads + writes here can become leader can become leaderRules:
replication.factor=3— recommended for production (tolerates 2 broker failures)- If the leader fails, Kafka elects a new leader from the ISR set only
min.insync.replicas— minimum ISR count before a write is acknowledged (set to 2 with RF=3)- Flow: producer → leader (disk write) → followers fetch → ISR updated
Quick Reference
TL;DR
A Kafka topic is split into ordered partitions. Each message gets a unique offset within its partition. Consumers track their position using committed offsets. Brokers store partitions — each partition has exactly one leader. Replicas (RF=3 recommended) keep data safe across brokers. If a leader fails, Kafka promotes a new one from the ISR. In Kafka 4, all cluster metadata is managed by KRaft — no ZooKeeper needed.
Support me through GitHub Sponsors.
Next
➡️ Day 3 — KRaft deep dive: controller quorum, Raft consensus, metadata log
Resources
- 📘 Kafka: The Definitive Guide — Chapter 1 & 2
- 🌐 kafka.apache.org/43/documentation.html
- 🌐 kafka.apache.org/43/intro
