60-Day Kafka 4 Learning Plan · Week 9 — Day 59 of 60

60-Day Kafka 4 Learning Plan · Week 9 — Capstone & Career Resources: Kafka: The Definitive Guide (all chapters) · kafka.apache.org/43/documentation

Goal

Walk through the interview scenarios and design questions that come up for senior Kafka engineers. Know how to reason through architecture decisions, KRaft internals, and operational problems — not just recite facts — including fixing a recurring piece of bad advice that showed up in Day 57 too, and reconciling an apparent contradiction in controller election timing.

1. Architecture scenario — design a real-time order system

Prompt: “Design a system where orders from a REST API reach a mobile app in under 1 second.”

Model answer — hit these 5 points

Producer: REST API publishes to labs.events using KafkaTemplate. Key = orderId/userId → same partition → ordered per user. acks=all + enable.idempotence=true = zero data loss.
Topic design: labs.events, partitions = 3 × broker count, RF=3, min.insync.replicas=2. For low latency: linger.ms=5 (not 20ms batch mode).
Consumer: @KafkaListener(concurrency=partitions) fans out to WebSocket via SimpMessagingTemplate → /topic/orders/{userId}.
Latency budget: produce ≈ 5ms · Kafka propagate ≈ 5ms · WS push ≈ 10ms → < 50ms total (well under 1s).
Observability & scale: KEDA ScaledObject on consumer lag. Grafana alert on lag > 1,000. Prometheus kafka_consumergroup_lag is the north-star metric.

A strong candidate also mentions what a weaker answer omits: point 3’s /topic/orders/{userId} needs an authenticated WebSocket handshake and convertAndSendToUser routing (Day 20 §8, and the gap caught concretely in Day 58 §7) — a client-guessable destination path is a real security omission worth naming even in a whiteboard-level design answer, not just an implementation detail to skip over.

2. KRaft-specific Q&A

Q: How does KRaft replace ZooKeeper?

A: Brokers elect a KRaft controller via Raft consensus. The active controller stores all cluster metadata (topic configs, partition assignments, ISR, leader epochs) in the @metadata topic — a replicated Kafka log. No external ZooKeeper process is needed. The @metadata topic is replicated to all controller voters for HA.

Q: What is the @metadata topic?

A: An internal Kafka topic that stores all cluster state. It is a compacted, replicated log maintained by the KRaft quorum. Brokers subscribe to changes from the active controller and cache metadata locally. This is how Kafka 4 achieves faster metadata propagation than ZooKeeper.

Q: Why does the KRaft quorum need an odd number of controllers?

A: Raft requires a majority quorum (n/2 + 1):

3 controllers → 2 must agree → tolerates 1 failure
5 controllers → 3 must agree → tolerates 2 failures
2 controllers → both must agree → 0 fault tolerance

Always deploy 3 or 5 controllers in production. Never 2 or 4.

Q: Can a node be both broker and controller in KRaft?

A: Yes — process.roles=broker,controller enables combined mode. Good for dev and small clusters. In production (Strimzi), use separate KafkaNodePool resources so broker and controller pools can scale independently and be placed on different node types.

Q: How does KRaft recover after the active controller crashes?

A: See §7 — the “5–30 seconds” figure below needs reconciling against Day 8’s “milliseconds” framing, and a strong interview answer should be able to explain both numbers, not just quote one.

The remaining quorum voters hold a Raft election. The voter with the most up-to-date log becomes the new leader. Brokers detect the change via metadata fetch timeout and reconnect to the new controller. Typical election time: 5–30 seconds. No manual intervention needed.

Q: What happens to producers during a KRaft controller election?

A: Producers keep sending. Brokers continue to accept produce requests using their locally cached partition leader info. Only metadata-changing operations (create topic, add partition, etc.) are temporarily unavailable during the election window.

3. “How would you…” design decisions

4. Tricky scenario Q&A

Q: Consumer lag is growing. What do you check first?

A: Reason through whether it’s producer-side or consumer-side:

Is it all partitions or just some? If some → key skew (too many events for one key, all routing to one partition). Fix: better key distribution or more partitions.
Has the producer rate spiked? Check kafka_server_broker_topic_metrics_messages_in_total rate. If yes → add consumers (up to partition count).
Is the consumer processing slowly? Check consumer GC pauses, DB query latency, downstream service slowness. Increase max.poll.records or process records in parallel.
Add consumer instances. Auto-scale up to partition count — beyond that, add partitions (note: you cannot decrease them later).

Q: Producer gets NotEnoughReplicasException. How do you respond?

This answer needs correcting — see §6. Step 3 below repeats the exact bad advice already fixed in Day 57 §7.

A: ISR has shrunk below min.insync.replicas. Step through: 1. Check UnderReplicatedPartitions metric — which brokers are offline? 2. Is it a network partition (some brokers unreachable)? 3. Short-term (accepting risk): lower min.insync.replicas or use acks=1 temporarily 4. Proper fix: restore the failed broker, wait for ISR to recover, then restore min.insync.replicas

Q: You need to reprocess all events from exactly 7 days ago. How?

A: Kafka retains data within log.retention.hours — no external replay needed:

# Reset consumer group offset to a specific timestamp
kafka-consumer-groups.sh \
  --bootstrap-server broker:9092 \
  --group labs-api-group \
  --topic labs.events \
  --reset-offsets \
  --to-datetime 2026-08-19T00:00:00.000 \
  --execute

The consumer group resumes from that timestamp on next poll. Kafka maps the timestamp to the nearest offset per partition automatically.

Q: A topic has 3 partitions and you need more throughput. What do you do?

A: Increase partition count (you can go up, not down):

kafka-topics.sh --bootstrap-server broker:9092 \
  --alter --topic labs.events --partitions 6

Important caveat: key-based routing changes after a partition increase — a key that was in partition 1 may now land in partition 4. If ordering per key is critical, plan the increase carefully (or use a consistent hashing scheme that pre-accounts for the new count).

Q: How would you design for exactly-once end-to-end?

A: Three layers:

Producer: enable.idempotence=true — no duplicate records at broker level
Consumer: manual commit after successful processing — enable.auto.commit=false
Kafka Streams: processing.guarantee=exactly_once_v2 — atomic read-process-write with transactions
External systems: use idempotent writes (upsert by eventId) on the sink side

Point 2 here has the same gap flagged in Day 57 §8: “manual commit after successful processing” alone gives at-least-once, not exactly-once. A complete interview answer should name the actual mechanism — idempotent downstream writes (which point 4 does cover) or, for a genuine consume-transform-produce pipeline, sendOffsetsToTransaction() tying the offset commit to the output write atomically (Day 11 §9).

5. The questions YOU should ask the interviewer

“What Kafka version are you running — are you on KRaft or still on ZooKeeper?”
“How do you handle consumer lag alerts today?”
“What is your current partition strategy — how was the count decided?”
“Do you have Schema Registry, or are you using plain JSON?”
“What is your RPO/RTO target for the Kafka cluster?”

Asking these shows you think operationally, not just architecturally.

6. Correcting the NotEnoughReplicasException answer — same bad advice as Day 57 §7

§4’s original answer to NotEnoughReplicasException recommends “lower min.insync.replicas or use acks=1 temporarily” as a short-term response — this is the identical anti-pattern already corrected in Day 57 §7, appearing again in a different context. Worth fixing consistently, since an interview answer that suggests downgrading durability under pressure is a genuine red flag a strong interviewer would catch.

The corrected answer:

NotEnoughReplicasException means ISR has shrunk below min.insync.replicas — with the standard RF=3/min.insync.replicas=2 setup, this specifically means two or more replicas are currently unavailable (losing just one is the normal, tolerated case — see Day 57 §7 for the full walkthrough). The correct response sequence:

1. Check UnderReplicatedPartitions and identify which brokers are offline or unreachable.

2. Determine root cause — broker crash, disk full (Day 55 §2), network partition.

3. Restore the failed broker(s) — this is the actual fix, following the broker-failure runbook (Day 55 §3).

4. Once ISR recovers to meet min.insync.replicas, writes resume automatically — no config change needed at any point.

5. Never lower min.insync.replicas or acks as a response to this exception — doing so doesn’t fix the underlying availability problem, it just silently accepts the exact data-loss risk the setting exists to prevent, and it’s easy to forget to revert afterward, leaving the cluster permanently less durable than intended.

A strong interview answer distinguishes “the system is telling me exactly what it’s designed to tell me under replica loss” from “the system is misconfigured and needs a durability downgrade” — the correct response to NotEnoughReplicasException is always incident recovery, never a config change to make the exception stop happening.

7. Reconciling controller election timing — “milliseconds” vs “5–30 seconds”

Day 8 §3 described KRaft controller failover as completing “in milliseconds,” contrasted against ZooKeeper’s 30–60 second recovery. This document’s §2 states “typical election time: 5–30 seconds” for the same event. Both are correct, describing different parts of the same recovery sequence — a strong interview answer explains the distinction rather than just picking one number to quote.

Total observed recovery time = failure detection + Raft vote + broker reconnection

Failure detection: the remaining quorum voters must first notice the active
  controller is actually gone — this relies on missed heartbeats/session-style
  timeouts, which take real seconds to trigger conservatively (avoiding a
  false-positive election on a brief blip)

Raft vote itself: genuinely fast — milliseconds — once triggered, since it's
  just quorum voters exchanging messages over an already-established
  replicated log, with no external ZooKeeper session negotiation

Broker reconnection: brokers detect the new controller via their own
  metadata fetch and reconnect — adds a further small delay

The accurate framing for an interview answer: “the Raft consensus mechanism itself is fast — milliseconds — which is the specific comparison point against ZooKeeper’s old session-timeout-based recovery. But the end-to-end observed recovery time, including detecting the failure in the first place, is more realistically several seconds to tens of seconds, matching real-world KRaft cluster behavior.” Being able to explain why two seemingly contradictory numbers both show up in different sources (marketing-oriented “milliseconds” framing vs operational “5–30 seconds” framing) is itself a signal of deeper understanding than memorizing either number alone.

8. Common pitfalls

Suggesting a durability downgrade (acks=1, lower min.insync.replicas) as a response to NotEnoughReplicasException — the exact anti-pattern corrected in both Day 57 §7 and here; the correct response is always incident recovery (§6)
Describing “manual commit after processing” as sufficient for exactly-once in a design-question answer — an interviewer probing deeper will ask what happens on a crash between processing and commit, and “at-least-once” is the honest answer without idempotent writes or transactional offset commits (§4, Day 57 §8)
Quoting only one of “milliseconds” or “5–30 seconds” for controller failover without being able to explain both — a strong answer distinguishes the fast Raft vote itself from the slower end-to-end observed recovery including failure detection (§7)
Describing a design’s WebSocket layer without mentioning authentication — an architecture answer that gets partition/RF/acks right but glosses over /topic/{userId}-style client-guessable destinations misses a real gap covered explicitly in Day 20 §8 and Day 58 §7 (§1)
Treating “how would you design X” answers as a checklist to recite rather than a chance to name explicit trade-offs (throughput vs latency, durability vs availability) — interviewers are generally listening for the reasoning, not just the vocabulary

Key Takeaways

In design questions: always mention key, partitions, RF, acks, security, and observability — including WebSocket/API-layer authentication, not just the Kafka-internal config
KRaft = Raft consensus stored in @metadata topic — no ZooKeeper, faster metadata ops; the Raft vote itself is millisecond-fast, but end-to-end observed controller failover (including failure detection) is more realistically seconds
For lag: distinguish producer spike vs slow consumer — different fixes for each
NotEnoughReplicasException means restore the failed broker(s), never lower acks/min.insync.replicas as a workaround — that’s Day 57 §7’s golden-rule lesson, worth remembering consistently across every context it comes up in
“Manual commit after processing” alone is at-least-once, not exactly-once — a complete answer names idempotent writes or transactional offset commits
Kafka retains data — reprocessing from history is a reset-offsets command, not an ETL job
Always explain the trade-off: throughput vs latency, durability vs availability — and be ready to explain why two numbers from different sources (like controller election timing) both appear correct rather than picking one to memorize

Support me through GitHub Sponsors.

Thank you for Reading !! See you in the next post.

➡️ Day 60: Kafka roadmap — Flink, Pulsar, Conduktor, what’s next?

Resources

📘 Kafka: The Definitive Guide — all chapters
🌐 kafka.apache.org/43/documentation