60-Day Kafka 4 Learning Plan · Week 2 — Day 10 of 60

---

60-Day Kafka 4 Learning Plan · Week 2 — Kafka Internals Sources: Kafka: The Definitive Guide Ch.5 · kafka.apache.org/43

Goal

Understand how Kafka replicates data across brokers, how leaders and followers interact, what the ISR is, how leader election works in KRaft, how to place replicas across racks/AZs, and how to configure and monitor durability correctly in Spring Boot.

1. Replica roles: leader, follower, ISR

Producer ──► Broker 1 (LEADER P0) ──┬──► Broker 2 (Follower · ISR)  ──► Consumer
                                    └──► Broker 3 (Follower · ISR)
                                          (dashed = fetch/pull model)

Leader: handles all reads and writes for a partition. One leader per partition.

Followers: replicate by continuously fetching from the leader (pull model, not push). They never serve reads by default (follower reads available from Kafka 2.4+ with replica.selector.class).

ISR (In-Sync Replicas): the subset of replicas that are fully caught up with the leader within replica.lag.time.max.ms (default 30s). The leader tracks ISR membership.

• ISR shrinks when a follower falls behind or crashes.
• ISR expands when the follower catches up again.
• A write is committed only when all ISR replicas have acknowledged it (with acks=all).

2. Producer write path — acks setting

acks=0 — fire and forget

• No acknowledgement from the broker.
• Highest throughput, lowest durability.
• Risk: messages silently lost if the broker crashes during write.
• Use for: non-critical metrics, logs, click events.

acks=1 — leader ACK only (default)

• Leader writes to its local log and ACKs the producer.
• Risk: message lost if the leader crashes before any follower fetches it.
• Use for: moderate durability, acceptable loss window.

acks=all (-1) — all ISR must ACK ★ recommended

• Leader waits for every replica in the ISR to confirm the write before ACKing.
• Strongest durability guarantee available.
• Must pair with min.insync.replicas to be meaningful.
• Use for: financial transactions, order events, anything you can’t lose.

3. min.insync.replicas

With acks=all, the broker refuses writes if fewer than min.insync.replicas replicas are in the ISR. This prevents silent data loss when the ISR shrinks.

Recommended production setup

replication.factor = 3
min.insync.replicas = 2
acks = all

• Tolerates 1 broker failure while guaranteeing writes land on at least 2 replicas.
• If only 1 ISR remains → producer receives NotEnoughReplicasException → can retry safely with idempotence enabled.
• If min.insync.replicas = 1 (default), acks=all only waits for the leader — no real protection.

4. Leader election flow

Leader fails          Controller detects       Elects new leader        Resume serving
(Broker 1 crashes) → (KRaft watches ISR)  → (First ISR replica wins) → (No data loss)

In Kafka 4 KRaft:

• The active controller is elected via Raft consensus — no ZooKeeper.
• The controller holds the full cluster metadata log (__cluster_metadata topic).
• Leader changes are propagated to all brokers immediately via the metadata log.
• Election is faster and more predictable than the old ZooKeeper-based approach.

Which replica becomes the new leader?

• The first replica in the ISR list (the “preferred leader” for that partition).
• If unclean.leader.election.enable=false (default), only ISR members are eligible.

5. Preferred vs unclean leader election

Preferred leader election (default)

• New leader is chosen from the ISR only.
• Since all ISR replicas are fully caught up → zero data loss guaranteed.
• unclean.leader.election.enable=false (default, keep it).

Unclean leader election

• An out-of-ISR replica is allowed to become leader.
• That replica may be missing messages the old leader had committed → data loss.
• Trades durability for availability — useful when all ISR members are down and downtime is unacceptable.
• unclean.leader.election.enable=true (only for non-critical topics).

Decision guide:

6. ISR shrink & expand

replica.lag.time.max.ms = 30 000 (default)

If ISR drops to 1 and min.insync.replicas=2 → broker throws NotEnoughReplicasException → producer retries. No silent data loss.

7. Follower fetch protocol & the high watermark

Followers don’t get pushed data — they run their own loop calling the Fetch API against the leader, the same protocol consumers use. This is why replication and consumption share the same performance characteristics and tooling.

Follower → Fetch(topic=orders, partition=0, offset=1000) → Leader
Leader   → returns records from offset 1000 + its current high watermark → Follower
Follower appends records, advances its own log end offset (LEO)
Follower's next Fetch request implicitly ACKs everything up to its new LEO

High Watermark (HW): the highest offset that has been written to all ISR replicas. Consumers can only read up to the HW — never beyond it, even if the leader’s local log has newer, not-yet-replicated records. This is what makes acks=all reads consistent: a consumer never sees a message that could still be lost if the leader crashes.

Leader log:     [0 ... 1000] ← leader's Log End Offset (LEO)
Follower 2 LEO: [0 ... 998]
Follower 3 LEO: [0 ... 1000]
High Watermark = min(LEO across ISR) = 998   ← consumers can read up to here only

Why this matters for latency: the slowest ISR member sets the HW for the whole partition. A single lagging-but-still-in-ISR follower directly increases end-to-end read latency for every consumer, even though it hasn’t been kicked out of the ISR yet.

8. Rack / AZ awareness — replica placement

By default, Kafka spreads replicas across brokers but has no concept of physical failure domains unless told. In cloud deployments, an entire availability zone can fail — if all 3 replicas of a partition happen to land in the same AZ, replication.factor=3 gives zero real protection against that AZ going down.

# server.properties — tag each broker with its physical location
broker.rack=us-east-1a

With broker.rack set on every broker, Kafka’s replica placement algorithm actively spreads replicas across racks/AZs, not just across brokers — so a single AZ failure can, at most, remove one replica from any given partition’s ISR.

Production checklist: always set broker.rack in cloud deployments with 3+ AZs. It costs nothing at write time and is the difference between “AZ outage = business as usual” and “AZ outage = degraded ISR everywhere.”

9. Partition reassignment

When adding brokers, decommissioning old ones, or fixing an unbalanced cluster (some brokers holding way more partitions/leader load than others), replicas need to move.

# 1. Generate a reassignment plan (dry run)
kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \
  --topics-to-move-json-file topics.json \
  --broker-list "1,2,3,4" \
  --generate

# 2. Execute the plan
kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json \
  --execute

# 3. Verify progress
kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json \
  --verify

Reassignment is essentially “add the new replica, let it fully catch up via the fetch protocol, then drop the old one” — so it consumes real network/disk I/O and should be throttled in production:

--throttle 50000000   # cap reassignment traffic at 50 MB/s so it doesn't starve live traffic

10. Follower reads (KIP-392)

Since Kafka 2.4, consumers can read from the nearest replica instead of always the leader — useful for multi-region deployments where reading from a same-region follower avoids expensive cross-region network hops.

# Broker side
replica.selector.class=org.apache.kafka.common.replica.RackAwareReplicaSelector

# Consumer side — must also set broker.rack matching the consumer's own location
spring.kafka.consumer:
  properties:
    client.rack: us-east-1a

Trade-off: follower reads may return slightly stale data relative to the leader (bounded by replication lag), since followers apply the HW asynchronously. Fine for read-heavy, latency-sensitive use cases; not appropriate where consumers need the absolute latest committed offset.

11. Monitoring replication health

# Check for unbalanced or under-replicated partitions directly
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --under-replicated-partitions

Alert priority: UnderMinIsrPartitionCount > 0 is a P1 — it means producers with acks=all are actively failing writes on those partitions right now, not just a future risk.

12. Common pitfalls

• replication.factor=3 without broker.rack in a multi-AZ cloud setup — all replicas can silently land in one AZ, negating the whole point of RF=3
• min.insync.replicas=1 with acks=all — durability theater; this only waits for the leader, exactly like acks=1
• Ignoring ISR churn — flapping ISR membership (shrink/expand repeatedly) usually signals a broker that’s undersized or a network issue, not a one-off blip
• Un-throttled partition reassignment — moving many partitions at once during business hours can saturate broker network and degrade live traffic
• Follower reads for strict-consistency use cases — client.rack reads can lag the leader; don’t use for workloads needing read-your-writes guarantees

13 Key configuration reference

14. Spring Boot: producer config for durability

spring:
  kafka:
    producer:
      acks: all                    # wait for all ISR
      retries: 3                   # retry on transient failure
      enable-idempotence: true     # exactly-once on retry (prevents duplicates)
    bootstrap-servers: localhost:9092

Topic-level override via NewTopic bean:

return TopicBuilder.name("orders")
        .partitions(3)
        .replicas(3)
        .config(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, "2")
        .build();

Why enable-idempotence=true? When a producer retries after a transient failure, it could send a duplicate if the broker received the first attempt but the ACK was lost. Idempotence assigns each message a sequence number — the broker deduplicates automatically. Required for exactly-once producer semantics.

Key Takeaways

• Leader handles all reads & writes; followers pull and replicate via the same Fetch API consumers use
• The high watermark (min LEO across ISR) bounds what consumers can read — the slowest ISR member sets the pace for everyone
• ISR = replicas fully caught up; shrinks on lag, expands on recovery
• acks=all + min.insync.replicas=2 + replication.factor=3 = production durability golden rule
• broker.rack is essential in multi-AZ clouds — without it, RF=3 can still fail entirely with one AZ outage
• KRaft controller manages leader election via Raft — no ZooKeeper in Kafka 4
• Unclean election = availability over durability — only for non-critical topics
• Follower reads (KIP-392) cut cross-region latency but trade off strict read-your-writes consistency
• enable-idempotence=true prevents duplicate writes on producer retry

---

If you loved reading the story, don’t forget to clap 👏. You can reach out to me and follow me on Medium, Twitter, GitHub, Linkedln

Support me through GitHub Sponsors.

Next

➡️ Day 11: Delivery Semantics

Resources

• 📘 Kafka: The Definitive Guide — Chapter 5 (Kafka Internals)
• 🌐 kafka.apache.org/43/documentation — replication & ISR config properties
• 🌐 Spring Kafka — Producer configuration
• 🌐 KIP-392: Allow consumers to fetch from closest replica

👉 Link to Medium blog