60-Day Kafka 4 Learning Plan · Week 1 — Foundations
Sources: Kafka: The Definitive Guide Ch.1 · kafka.apache.org/43
Goal
Understand what Kafka is, why it was created, how it differs from queues & databases, and where it fits in real systems.
1. The problem Kafka was built to solve
LinkedIn (2010) had two disconnected systems:
- A metrics system — polling-based, coarse granularity, XML batches
- An activity tracker — hourly batch jobs, fragile schemas
Neither could feed the other. Every new consumer required a new point-to-point connection. The architecture became spaghetti.
The insight — Jay Kreps, Neha Narkhede, Jun Rao:
“Instead of focusing on holding piles of data like our relational databases… we would focus on treating data as a continually evolving and ever growing stream.”
2. What Kafka actually is
Apache Kafka = a distributed event streaming platform
It lets you:
- Publish streams of events (producers)
- Subscribe to streams of events (consumers)
- Store events durably with configurable retention
- Process events in real time or retrospectively
Core data model
Producer A ─┐
Producer B ─┼──► Topic: "orders" [Partition 0][Partition 1][Partition 2]
Producer C ─┘ │
├──► Consumer Group A (own offset)
├──► Consumer Group B (own offset)
└──► Consumer Group C (own offset)Key concepts
3. Kafka vs queues vs databases
💡 Your stack: Kafka replaces RabbitMQ when you need replay, multiple independent consumers, or >100k msg/sec throughput.
4. Real-world use cases
5. Why Kafka 4? — KRaft in 60 seconds
Before (Kafka 3.x) — ZooKeeper required
Broker 1 ─┐
Broker 2 ─┼──► ZooKeeper ensemble ← separate system, extra failure domain
Broker 3 ─┘After (Kafka 4) — KRaft only ✓
┌─────────────────────────────────────────────────────────────┐
│ One Kafka cluster — metadata managed internally via Raft │
│ │
│ [Broker 1] [Broker 2] [Broker 3] [Controller quorum] │
│ Kafka+KRaft Kafka+KRaft Kafka+KRaft Raft consensus │
└─────────────────────────────────────────────────────────────┘KRaft benefits:
- ✓ No ZooKeeper to deploy, monitor, or fail
- ✓ Faster startup and controller failover
- ✓ Simpler ops — one system instead of two
- ✓ Scales to millions of partitions
- ✓ Kafka 4 is KRaft-only — ZooKeeper is completely removed
TL;DR
Kafka was born at LinkedIn to unify metrics + activity tracking into one durable, high-throughput, replayable event stream. Unlike queues (messages disappear after consumption) or databases (not designed for high-throughput streaming), Kafka stores events in ordered, partitioned logs that multiple independent consumers can read at their own pace. Kafka 4 removes ZooKeeper entirely — KRaft is now the metadata backbone.
If you loved reading the story, don’t forget to clap 👏. You can reach out to me and follow me on Medium, Twitter, GitHub, Linkedln
Support me through GitHub Sponsors.
Next
➡️ Day 2 — Core concepts: topics, partitions, offsets & replicas in depth
Resources
- 📘 Kafka: The Definitive Guide — Chapter 1 (Neha Narkhede, Gwen Shapira, Todd Palino)
- 🌐 kafka.apache.org/43/documentation.html
- 🌐 kafka.apache.org/43/getting-started/introduction
