60-Day Kafka 4 Learning Plan · Week 9 — Day 57 of 60

60-Day Kafka 4 Learning Plan · Week 9 — Capstone & Career Resources: Kafka: The Definitive Guide (all chapters) · kafka.apache.org/43

Goal

Consolidate everything from the previous 56 days into a single reference. Revisit the key concept from each week, lock in the must-know production configs, memorise the critical alert thresholds, work through the interview-trap gotchas — including fixing a genuine error in one of them that contradicts the course’s own core teaching from Day 10 — and step back to name the handful of patterns that recurred across the entire 8 weeks.

1. 8-week journey summary

2 . Must-know configs: production defaults

Producer — high throughput + safety

acks=all                                  # wait for all ISR replicas
enable.idempotence=true                   # exactly-once delivery
batch.size=65536                          # 64 KB batch
linger.ms=20                             # fill batch window
compression.type=snappy                   # reduce network bytes
retries=2147483647                        # retry until delivery.timeout.ms
max.in.flight.requests.per.connection=5   # pipeline (safe with idempotence)
delivery.timeout.ms=120000               # 2-minute overall timeout
security.protocol=SASL_SSL               # encrypt + authenticate

Per Day 52 §9, prefer compression.type=zstd over snappy as the general default unless a specific client-compatibility constraint applies.

Consumer — reliable processing

enable.auto.commit=false                  # manual commit for reliability
auto.offset.reset=earliest               # don't miss messages on new groups
fetch.min.bytes=1048576                   # 1 MB min fetch
max.poll.records=500                      # records per poll
max.poll.interval.ms=300000              # 5 min before considered dead
session.timeout.ms=45000                 # heartbeat timeout

Per Day 52 §7, fetch.min.bytes=1MB is only appropriate for genuinely high-throughput topics — applying it uniformly to a low-volume topic adds latency for no benefit.

Broker — durability + performance

default.replication.factor=3
min.insync.replicas=2
log.retention.hours=168                   # 7 days
num.network.threads=8
num.io.threads=16
authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
super.users=User:admin
sasl.enabled.mechanisms=SCRAM-SHA-512
KAFKA_JMX_PORT=9999
KAFKA_HEAP_OPTS=-Xms6g -Xmx6g            # fixed JVM heap

3 . Alert on these: never ignore

The “consumer lag > 1,000” row is exactly the absolute-threshold trap Day 48 §6 warned against — a single number applied uniformly means very different things for a high-throughput topic (trivial) vs a low-throughput one (potentially severe). Treat this row as a starting-point default, not a universal constant — prefer the time-to-drain normalization or per-group thresholds from Day 48 §6 once you have real topics to tune against.

4 . Common gotchas: interview traps

Q: Can you have more consumers than partitions in a consumer group?

A: Yes — but extra consumers are idle. They get no partition assigned. You cannot exceed partition count for meaningful parallelism.

Q: What happens if min.insync.replicas=2 and one replica goes down?

This answer needs correcting — see §7. The version below is what §4 originally stated, and it contradicts Day 10 §3’s own golden rule.

A: Produces fail with NotEnoughReplicasException. This is by design — Kafka refuses writes rather than risk silent data loss. Increase acks=1 or accept reduced durability to work around it.

Q: Can you decrease partition count after creation?

A: No — Kafka only allows increasing partitions. Decreasing is not supported. Plan partition count carefully from the start (use 3× broker count as a baseline).

Q: Are offsets preserved across topics with the same name in different clusters?

A: No — offsets are cluster-local. The same record may have different offsets on primary vs backup. Use MirrorMaker 2’s offset translation (RemoteClusterUtils) to map offsets on failover.

Q: Does Kafka guarantee ordering of messages?

A: Only within a partition. Across partitions — no ordering guarantee. Route related events to the same partition using a consistent key.

Q: What’s the difference between PLAINTEXT, SSL, and SASL_SSL listeners?

PLAINTEXT — no encryption, no auth. Internal cluster traffic only.
SSL — TLS encryption + optional mTLS identity. No credential-based auth.
SASL_SSL — SASL credential authentication over TLS. Production standard.
SASL_PLAINTEXT — SASL auth with NO encryption. Never use in production.

Q: What is the KRaft quorum and why does it need an odd number of controllers?

A: The KRaft Raft consensus requires a majority quorum to elect a leader. With 3 controllers, 2 must agree (tolerates 1 failure). With 2, both must agree (no fault tolerance). Always use 3 or 5 controllers in production.

Q: When does a consumer group rebalance trigger?

A: A rebalance triggers when: a consumer joins/leaves the group, a consumer fails its heartbeat (session.timeout.ms exceeded), partition count changes, or subscriptions change. Rebalances pause consumption — minimise them with static membership (group.instance.id).

Q: What is exactly-once semantics in Kafka?

A: End-to-end exactly-once requires all three — see §8 for a refinement of point 2, which understates what’s actually required:

Producer: enable.idempotence=true + acks=all — no duplicate records at broker
Consumer: manual commit after processing — no double-processing
Kafka Streams / Transactions: processing.guarantee=exactly_once_v2 for Streams apps

5. Key CLI commands you should know cold

# Topic management
kafka-topics.sh --bootstrap-server broker:9092 --create --topic labs.events --partitions 6 --replication-factor 3
kafka-topics.sh --bootstrap-server broker:9092 --describe --topic labs.events
kafka-topics.sh --bootstrap-server broker:9092 --describe --under-replicated-partitions

# Consumer groups
kafka-consumer-groups.sh --bootstrap-server broker:9092 --list
kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group labs-api-group
kafka-consumer-groups.sh --bootstrap-server broker:9092 --reset-offsets --to-latest --group labs-api-group --all-topics --execute

# SCRAM users (SASL)
kafka-configs.sh --bootstrap-server broker:9092 --alter \
  --add-config 'SCRAM-SHA-512=[iterations=8192,password=secret]' \
  --entity-type users --entity-name labs-api

# ACLs
kafka-acls.sh --bootstrap-server broker:9094 --list
kafka-acls.sh --bootstrap-server broker:9094 --add \
  --allow-principal User:labs-api --operation Write --topic labs.events

# Performance test
kafka-producer-perf-test.sh --topic labs.events --num-records 1000000 \
  --record-size 1024 --throughput -1 \
  --producer-props bootstrap.servers=broker:9092 batch.size=65536 linger.ms=20

# Leader election (after broker recovery)
kafka-leader-election.sh --bootstrap-server broker:9092 \
  --election-type preferred --all-topic-partitions

6. Architecture decision cheat sheet

7. Correcting the min.insync.replicas Q&A — it contradicts Day 10’s golden rule

The original §4 answer to “what happens if min.insync.replicas=2 and one replica goes down” states that produces fail with NotEnoughReplicasException and suggests lowering acks=1 to work around it. This is backwards, and it directly contradicts the golden rule established in Day 10 §3.

RF=3, min.insync.replicas=2, acks=all

ONE replica goes down → ISR shrinks from [leader, follower-A, follower-B] to [leader, follower-A]
                       → ISR size = 2, which STILL MEETS min.insync.replicas=2
                       → writes continue succeeding normally — this is the golden rule
                         working exactly as designed, not a failure state

TWO replicas go down → ISR shrinks further to [leader] alone
                      → ISR size = 1, which is BELOW min.insync.replicas=2
                      → NOW writes fail with NotEnoughReplicasException

The corrected answer:

With RF=3 and min.insync.replicas=2, losing one replica is exactly the scenario this configuration is designed to tolerate transparently — ISR still has 2 members, writes continue succeeding, and no application-visible error occurs. NotEnoughReplicasException only fires when ISR drops below min.insync.replicas — i.e., when a second replica is also lost (or the broker holding it is unreachable). When that does happen, the correct response is not to lower acks — doing so defeats the entire purpose of the setting and reopens exactly the silent-data-loss risk min.insync.replicas exists to prevent. The correct response is to treat it as an active availability incident: follow the broker-failure runbook (Day 55 §3) to restore ISR, and let the producer’s configured retries (idempotence-safe, Day 16) continue attempting delivery until ISR recovers.

Why this error is worth dwelling on rather than just quietly fixing: this is a genuinely common real-world misconception, and “just lower acks when you hit NotEnoughReplicasException” is exactly the kind of advice that sounds reasonable under production pressure but actively undermines the durability guarantee the team presumably configured min.insync.replicas=2 to get in the first place. Recognizing that losing one replica at RF=3/MIR=2 is normal, tolerated operation — not an error state — and that losing a second replica should trigger an incident response rather than a config downgrade, is the actual interview-worthy distinction here.

8. Refining the exactly-once consumer-side answer

§4’s exactly-once answer states “Consumer: manual commit after processing — no double-processing” as point 2 — this is necessary but not sufficient, and understates what Day 11 actually covered.

What “manual commit after processing” alone actually gives you: at-least-once delivery (a crash between processing and commit causes reprocessing on restart), not exactly-once — this is exactly the distinction Day 11 §3 draws. Manual commit ordering avoids the worst case (auto-commit losing track of unprocessed work), but by itself doesn’t prevent duplicate processing on a crash-and-restart. Genuine exactly-once on the consume side requires either the consumer’s downstream writes being idempotent (deduping by message key/ID, Day 18 §5’s replay-idempotency principle applied here too), or — for a full consume-transform-produce pipeline — the transactional pattern from Day 11 §9: sendOffsetsToTransaction() tying the input offset commit to the output produce as one atomic unit, so a crash mid-way means neither happened, not that the input was silently reprocessed against an already-emitted output.

9. The patterns that recurred across this entire course — a meta-summary

Stepping back across all 56 days: a handful of specific mistakes showed up repeatedly, in different technologies and different weeks, often because the same underlying trade-off gets rediscovered independently each time a new component is introduced. Worth naming explicitly as a checklist, since recognizing the pattern is more durable than remembering any single day’s specific fix.

Why this list matters more than any individual entry in it: the specific technology changes every week — RocksDB state stores, Connect internal topics, Redis Pub/Sub, Schema Registry — but the underlying trade-off categories are small in number and repeat constantly. A engineer who internalizes “check replication factor,” “check for auth on any new admin surface,” “distrust absolute alert thresholds,” “ask what happens with 2+ instances,” “never let a credential sit in plaintext config,” and “a config that looks right isn’t a tested plan” as standing habits will catch the next technology’s version of these same mistakes without needing to have seen that specific technology before.

10. Common pitfalls

Treating “produce fails with NotEnoughReplicasException” as the normal outcome of losing one replica at RF=3/MIR=2 — it’s the outcome of losing a second replica; one loss is the tolerated, designed-for case (§7)
“Lower acks to work around NotEnoughReplicasException“ — actively undermines the durability guarantee the configuration was set up to provide; the correct response is incident recovery, not a config downgrade (§7)
Treating “manual commit after processing” as sufficient for exactly-once — it gives at-least-once; genuine exactly-once needs idempotent downstream writes or the transactional sendOffsetsToTransaction pattern (§8)
Copying the “consumer lag > 1,000” threshold verbatim into a real alerting config — it’s a starting-point default, not a tuned number for any specific topic’s actual throughput (§3, Day 48 §6)
Reviewing each week’s material in isolation without stepping back for the cross-cutting patterns — the same handful of mistake categories (§9) recur across nearly every technology covered; recognizing the pattern generalizes better than memorizing each individual fix

Key Takeaways

KRaft replaces ZooKeeper — the biggest Kafka 4 change. No ZK pods needed.
Production safety triad: acks=all + min.insync.replicas=2 + RF=3 — and losing exactly one replica under this triad is normal, tolerated operation, not a failure state; only losing a second replica triggers NotEnoughReplicasException
Security layers: TLS (encrypt wire) + SASL (identity) + ACLs (authorisation)
OfflinePartitionsCount > 0 = data unreadable — page immediately, no exceptions
Ordering is per-partition only — use a consistent key to route related events together
You cannot decrease partition count — start with at least 3× broker count
“Manual commit after processing” is at-least-once, not exactly-once — genuine exactly-once needs idempotent consumers or transactional offset commits
A small set of mistake patterns (RF=1 left in examples, unauthenticated admin APIs, absolute alert thresholds, multi-instance state assumptions, plaintext credentials, untested “working” configs) recurred across nearly every week of this course — recognizing the pattern matters more than memorizing each individual instance

Support me through GitHub Sponsors.

Thank you for Reading !! See you in the next story.

➡️ Day 58: Capstone build — API → Kafka 4 → WebSocket → Android

Resources

📘 Kafka: The Definitive Guide — all chapters
🌐 kafka.apache.org/43