60-Day Kafka 4 Learning Plan · Week 8 — Day 51 of 60

60-Day Kafka 4 Learning Plan · Week 8 — Production & Cloud Sources: Kafka: The Definitive Guide Ch.4, 6 · kafka.apache.org/documentation/#brokerconfigs

Goal

Make data-driven decisions about how many partitions, what replication factor, how much disk, and how many brokers a production Kafka cluster needs — using formulas rather than guesswork, understanding the concrete mechanisms behind the “don’t over-partition” rule, and validating generic benchmark numbers against your own actual workload.

1. How many partitions?

Partitions are the unit of parallelism in Kafka. More partitions = more consumer threads = higher throughput. But too many partitions means more open file handles, longer leader elections after a failure, and more JVM heap for index tracking.

The formula

partitions = max(
  target_throughput_MB_s / producer_throughput_per_partition_MB_s,
  target_throughput_MB_s / consumer_throughput_per_partition_MB_s
)

Example:

Target: 500 MB/s write throughput
Producer throughput per partition: ~50 MB/s (typical on SSD)
Consumer throughput per partition: ~100 MB/s

partitions = max(500/50, 500/100) = max(10, 5) = 10 partitions

Round up to a multiple of your broker count for even distribution (e.g. 12 for a 3-broker cluster).

Partition sizing rules

Can I increase partitions later?

Yes — you can always add partitions to an existing topic:

kafka-topics.sh --bootstrap-server broker:9092 \
  --alter --topic labs.events --partitions 12

Warning: you cannot decrease partitions. Key-based ordering is only guaranteed within a partition — adding partitions reshuffles which partition a key maps to.

2. Replication factor — availability vs cost

The replication factor (RF) determines how many copies of each partition exist across brokers.

RF=3 (production standard) ★

Tolerates 1 broker failure with zero data loss
min.insync.replicas=2 + acks=all requires 2 replicas to acknowledge each write
Storage cost = 3× raw data size

# server.properties
default.replication.factor=3
min.insync.replicas=2

# producer config (labs-api)
acks=all

RF=2

Tolerates 1 broker failure IF min.insync.replicas=1
Risky: if the leader fails, the single follower may not be fully caught up
2× storage cost — a compromise that often isn’t worth it

RF=1 (dev only ⚠)

No redundancy — one broker failure loses all data in that partition
Fine for local dev and throwaway test environments
Never use in production

ISR and min.insync.replicas interaction

Producer write → Leader → ISR followers (RF-1 copies)

If ISR shrinks below min.insync.replicas → broker rejects writes
→ Producer gets NotEnoughReplicasException
→ Prefer this over silent data loss

3. Disk capacity formula

disk_per_broker_GB = (write_MB_s × retention_seconds × RF / broker_count) / 1024 × headroom_factor

headroom_factor = 1.2  (20% buffer for compaction, temp files, OS)

Worked example — labs-api

Write rate: 50 MB/s
Retention: 7 days = 604,800 seconds
Replication factor: 3
Broker count: 3
Headroom: 1.2×

disk_per_broker = (50 × 604800 × 3) / 3 / 1024 × 1.2
               = 90,720,000 / 3 / 1024 × 1.2
               = 29,531 GB × 1.2
               ≈ 35,437 GB ≈ 35 TB per broker

Retention config

# server.properties — time-based retention
log.retention.hours=168          # 7 days
log.retention.bytes=-1           # unlimited by size (time takes precedence)

# Or size-based retention (choose one)
log.retention.bytes=107374182400 # 100 GB per partition
log.retention.hours=-1

# Segment size (affects how quickly retention is enforced)
log.segment.bytes=1073741824     # 1 GB segments (default)

4. How many brokers?

Calculate from three independent constraints and take the maximum:

Per-broker throughput benchmarks (cloud VMs)

5. Sizing cheat sheet — labs-api scenario

# Topic: labs.events
# Write: 50 MB/s | Retention: 7d | RF: 3 | Brokers: 3

# Partition count
# max(50/50, 50/100) = 1 → round to 6 (2× broker count)
num.partitions=6

# Replication
default.replication.factor=3
min.insync.replicas=2

# Retention (7 days)
log.retention.hours=168
log.retention.bytes=-1
log.segment.bytes=1073741824

# Disk per broker ≈ 35 TB
# (provision 40 TB SSD with 15% spare for OS + logs)

# Create labs.events with correct sizing
kafka-topics.sh --bootstrap-server broker:9092 \
  --create \
  --topic labs.events \
  --partitions 6 \
  --replication-factor 3 \
  --config retention.ms=604800000 \
  --config min.insync.replicas=2

6. Why the 200k partition limit — the concrete mechanisms behind it

§1’s table states the hard limit without explaining what actually breaks — worth knowing the real mechanisms, since they’re what should drive whether your cluster’s safe ceiling is higher or lower than the generic 200k number.

Why this matters for the “3 × broker count as starting point” rule of thumb (§1): that heuristic is a reasonable default, not a law of physics — a cluster with genuinely high per-partition throughput needs might justify going well past it, while a cluster running many small, low-traffic topics can hit painful operational friction (slow controller failover, file handle pressure) at a much lower partition count than 200k if those topics accumulate across many teams. Treat the 200k figure as “start worrying and actively monitoring well before this,” not a green light up to that exact number.

7. Partition reassignment needs temporary extra disk headroom

§3’s disk formula sizes for steady-state retention — it doesn’t account for the temporary disk spike during a partition reassignment (Day 10 §9), which is a routine operational event (adding brokers, rebalancing after a failure) that this capacity plan needs to survive without running out of space.

Reassignment in progress: OLD replica + NEW replica both exist temporarily
  on their respective brokers until the new replica fully catches up
  → the broker gaining a new replica needs room for it ON TOP OF its
    existing steady-state allocation, however briefly

Practical sizing adjustment: provision meaningfully more headroom than just the 1.2× compaction/OS buffer in §3 — a broker that’s sized to exactly its steady-state disk need with no slack can run out of space mid-reassignment, turning a routine rebalancing operation into an incident. This is a good reason the worked example’s “40 TB SSD with 15% spare” (§5) should be treated as a floor, not a target — validate against your actual largest expected reassignment operation (e.g. adding one new broker and rebalancing a meaningful fraction of partitions onto it), not just static retention math.

8. Producer memory scales with partition count too

buffer.memory (Day 3/16’s producer accumulator) is allocated per producer instance, but its effective per-partition share shrinks as partition count grows — worth connecting the dots between this sizing exercise and producer-side configuration from Day 16, since the two are more related than they might appear in isolation.

buffer.memory (default 32MB) is shared across ALL partitions a producer writes to.
More partitions the same producer targets → less buffer per partition on average
→ can mean smaller batches per partition → less efficient batching (Day 3 §4)

Why this belongs in a sizing conversation, not just a producer-tuning one: choosing to scale partition count up (§1) without revisiting buffer.memory/batch.size on producers that write across many of those partitions can quietly degrade the batching efficiency (and therefore compression ratio, Day 13 §4) that was working well at a lower partition count. Partition count and producer buffer sizing are coupled decisions — when this sizing exercise significantly changes partition count, revisit producer buffer configuration in the same pass rather than treating them as unrelated tuning exercises done at different times.

9. Benchmark your own throughput — don’t trust generic per-partition numbers

§1’s “~50 MB/s producer, ~100 MB/s consumer per partition” and §4’s per-instance-type throughput table are useful starting points, but they’re exactly the kind of generic benchmark figures Day 13 §9 already warned against trusting blindly for compression — the same caution applies here, for the same reason: real payload shape, compression codec, and actual hardware all shift the real number meaningfully.

# Kafka ships purpose-built benchmarking tools — use them against YOUR
# actual message size/compression/replication settings, not generic figures
kafka-producer-perf-test.sh \
  --topic labs.sizing-test \
  --num-records 1000000 \
  --record-size 1024 \
  --throughput -1 \
  --producer-props bootstrap.servers=broker:9092 acks=all compression.type=zstd

kafka-consumer-perf-test.sh \
  --topic labs.sizing-test \
  --bootstrap-server broker:9092 \
  --messages 1000000

What to actually vary in the benchmark: message size and compression codec (Day 13) meaningfully change per-partition throughput — a workload of large, well-compressible JSON payloads behaves very differently from small, high-cardinality binary records, even on identical hardware. Run kafka-producer-perf-test.sh/kafka-consumer-perf-test.sh with your actual production message shape and acks/compression settings before finalizing the partition-count formula in §1 — the generic 50/100 MB/s figures are a starting estimate for the very first sizing pass, not a number to defend once real traffic is available to measure against.

10. Monitoring capacity trends — catching under-provisioning before it’s urgent

A sizing exercise is a point-in-time estimate; production traffic grows, and the useful habit is tracking actual usage against the plan continuously rather than re-deriving the formula only when something breaks.

Why this closes the loop on the whole exercise: every formula in this material (§1, §3, §4) produces a number based on current assumptions about throughput and retention — a growing system will eventually outgrow those assumptions, and the only way to know when is to keep measuring against the plan rather than treating the initial sizing exercise as a one-time calculation.

11. Common pitfalls

Treating “3 × broker count” or “200k partitions” as hard rules rather than starting heuristics — the real constraints (§6) are file handles, leader election time, and controller metadata size, which vary by actual workload, not a single universal number
Sizing disk for steady-state retention only, with no reassignment headroom — a routine rebalancing operation can run a tightly-provisioned broker out of space (§7)
Increasing partition count without revisiting producer buffer.memory/batch.size — silently degrades batching efficiency and compression ratio on producers writing across the newly-expanded partition set (§8)
Trusting generic per-partition throughput figures without benchmarking actual payload/compression settings — the same caution from Day 13 §9, applied here to capacity planning instead of codec selection (§9)
Treating a sizing exercise as a one-time calculation — without ongoing monitoring against the original assumptions, growth silently erodes the safety margin the plan was built with (§10)

Key Takeaways

Partitions = parallelism cap — size to throughput, round up to a multiple of broker count
The 200k partition guidance isn’t arbitrary — it’s a proxy for file handle pressure, leader election time, and controller metadata size, all of which scale with partition count
RF=3 + min.insync.replicas=2 is the production standard — tolerates 1 broker failure
Disk = write_rate × retention × RF / brokers × 1.2 — but add further headroom for partition reassignment spikes, not just steady-state compaction/OS overhead
Partition count and producer buffer.memory/batch.size are coupled — revisit both together when scaling partitions
Broker count = max(throughput constraint, storage constraint, availability = RF)
Benchmark your own workload’s actual per-partition throughput with kafka-producer-perf-test.sh/kafka-consumer-perf-test.sh — generic figures are a starting estimate, not a number to design around blindly
Monitor actual usage against the sizing plan’s original assumptions continuously — a sizing exercise is a snapshot, not a permanent guarantee
You can always increase partitions later, but you can never decrease them

Support me through GitHub Sponsors.

Thank you for Reading !! See you in the next post.

➡️ Day 52: Performance tuning — batch.size, linger.ms, buffer config

Resources

📘 Kafka: The Definitive Guide — Chapters 4 & 6
🌐 kafka.apache.org/documentation/#brokerconfigs
🌐 Kafka — Producer/Consumer performance testing tools

Link to Medium blog