60-Day Kafka 4 Learning Plan · Week 8 — Day 52 of 60

60-Day Kafka 4 Learning Plan · Week 8 — Production & Cloud Sources: Kafka: The Definitive Guide Ch.4, 5 · kafka.apache.org/documentation/#producerconfigs · #consumerconfigs

Goal

Squeeze maximum throughput out of Kafka producers and consumers without sacrificing durability. Tune producer batching, consumer fetch sizes, broker thread counts, and OS-level settings — understand the latency and durability trade-offs each tuning knob actually carries — then benchmark against a realistic topology to confirm real gains.

1. Producer tuning — batching is the key lever

Kafka producers accumulate records in a per-partition batch before sending. The batch is sent when either batch.size bytes are accumulated OR linger.ms milliseconds have elapsed — whichever comes first.

# application.yml — labs-api high-throughput producer
spring.kafka.producer:
  # Batching
  batch-size: 65536           # 64 KB (default: 16 KB)
  buffer-memory: 67108864     # 64 MB total accumulator buffer (default: 32 MB)

  properties:
    linger.ms: 20             # wait up to 20ms to fill a batch (default: 0 = send immediately)
    compression.type: snappy  # compress batch before network send (lz4 for max speed)
    max.in.flight.requests.per.connection: 5   # pipeline requests per broker connection
    acks: all                 # require all ISR replicas to acknowledge (durability)
    retries: 3
    retry.backoff.ms: 100
    delivery.timeout.ms: 120000

Compression type comparison

See §9 for a reconciliation with Day 13’s compression guidance — this table’s own numbers actually show zstd beating snappy on both dimensions, which is worth noticing before defaulting to the ★ marked here.

Producer perf test

# Baseline test — default settings
kafka-producer-perf-test.sh \
  --topic labs.events \
  --num-records 1000000 \
  --record-size 1024 \
  --throughput -1 \
  --producer-props bootstrap.servers=broker:9092

# Tuned test — compare throughput (MB/s and msg/s)
kafka-producer-perf-test.sh \
  --topic labs.events \
  --num-records 1000000 \
  --record-size 1024 \
  --throughput -1 \
  --producer-props bootstrap.servers=broker:9092 \
    batch.size=65536 \
    linger.ms=20 \
    compression.type=snappy

2. How batch.size and linger.ms interact

Default (linger.ms=0, batch.size=16KB) — low latency, low throughput

Record arrives → send immediately (batch rarely fills)
Result: 1 record per request → high request overhead
Use when: latency matters more than throughput (interactive apps)

Tuned (linger.ms=20ms, batch.size=64KB) — high throughput

Record arrives → wait up to 20ms OR until 64KB fills
Result: many records per request → 5-10× throughput gain
Use when: streaming, analytics, event pipelines

Key insight: idempotent producers and ordering

properties:
  enable.idempotence: true                      # exactly-once delivery guarantee
  max.in.flight.requests.per.connection: 5      # safe with idempotence enabled
  acks: all                                     # required when enable.idempotence=true

With enable.idempotence=true, Kafka deduplicates retried records — safe to set max.in.flight=5 for pipeline throughput without risking duplicates.

3. Consumer tuning — fetch and poll config

# application.yml — labs-socket high-throughput consumer
spring.kafka.consumer:
  # Fetch sizing — wait for meaningful chunks
  fetch-min-size: 1048576     # 1 MB minimum fetch (default: 1 byte)
  fetch-max-wait: 500         # ms to wait if fetch-min-size not yet available
  max-poll-records: 500       # max records returned per poll() call

  properties:
    fetch.max.bytes: 52428800        # 50 MB max per fetch response
    max.partition.fetch.bytes: 1048576   # 1 MB per partition per fetch

  listener:
    concurrency: 6            # listener threads — set equal to partition count

  # Commit strategy
  enable-auto-commit: false   # manual commit for exactly-once processing
  auto-offset-reset: earliest

Concurrency tuning

// @KafkaListener concurrency matches partition count
@KafkaListener(
    topics = "labs.events",
    groupId = "labs-socket-group",
    concurrency = "6"           // 6 threads for 6 partitions = max parallelism
)
public void consume(ConsumerRecord<String, String> record) {
    // each thread handles one partition
}

Consumer perf test

kafka-consumer-perf-test.sh \
  --bootstrap-server broker:9092 \
  --topic labs.events \
  --messages 1000000 \
  --fetch-size 1048576 \
  --threads 6

4. Broker OS & JVM tuning

Linux OS settings

# /etc/sysctl.conf — append and apply with: sysctl -p
net.core.rmem_max=134217728        # 128 MB receive socket buffer
net.core.wmem_max=134217728        # 128 MB send socket buffer
net.ipv4.tcp_rmem=4096 87380 134217728
net.ipv4.tcp_wmem=4096 65536 134217728
vm.swappiness=1                    # minimize swap — Kafka lives in page cache
vm.dirty_ratio=80                  # allow 80% of RAM to be dirty before flush
vm.dirty_background_ratio=5        # start background flush at 5%

# File descriptor limits (/etc/security/limits.conf)
kafka   soft    nofile  100000
kafka   hard    nofile  100000

vm.dirty_ratio=80 deserves a second look before copying it — see §8. It’s a real, deliberate throughput/durability trade-off, not a purely positive tuning win, and worth understanding before setting it that high.

Broker JVM

# KAFKA_HEAP_OPTS — fixed heap prevents GC resizing pauses
export KAFKA_HEAP_OPTS="-Xms6g -Xmx6g"

# GC: G1GC is recommended for Kafka (default in Java 11+)
export KAFKA_JVM_PERFORMANCE_OPTS="-XX:+UseG1GC \
  -XX:MaxGCPauseMillis=20 \
  -XX:InitiatingHeapOccupancyPercent=35 \
  -XX:+ExplicitGCInvokesConcurrent"

Rule: Give Kafka brokers 6–8 GB JVM heap and leave the rest of RAM for the OS page cache. A 32 GB broker node → 6 GB JVM, 26 GB page cache.

Broker server.properties

# I/O threads
num.network.threads=8          # threads for network requests (default: 3)
num.io.threads=16              # threads for disk I/O (default: 8)

# Socket buffers
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600   # 100 MB max request size

# Log flush (rely on OS page cache flush, not Kafka's internal flush)
log.flush.interval.messages=Long.MAX_VALUE
log.flush.interval.ms=Long.MAX_VALUE

Long.MAX_VALUE here is a Java identifier, not valid properties-file syntax — see §6 for the actual fix. A real server.properties file needs the literal numeric value, or this line either fails to parse or gets silently ignored depending on the tooling reading it.

5. Tuning quick-reference

6. Fixing the invalid config — `Long.MAX_VALUE` isn’t a value

log.flush.interval.messages=Long.MAX_VALUE / log.flush.interval.ms=Long.MAX_VALUE in §4 write out the name of a Java constant, not an actual numeric value a properties file parser understands. The intent (disable Kafka’s own periodic flush entirely, relying on the OS page cache and background flush instead) is a legitimate, common production setting — it just needs the real number.

# Corrected — the actual long value, not the Java identifier
log.flush.interval.messages=9223372036854775807
log.flush.interval.ms=9223372036854775807

Why this specific mistake is easy to make and easy to miss: Long.MAX_VALUE is exactly how Kafka’s own source code and documentation prose often refer to this setting’s effective default/disabled state — so copying that phrase directly into a config file feels natural, especially when skimming documentation rather than an actual working server.properties example. Any config value copied from prose explanation (rather than a verified working config file) is worth double-checking against the tool’s actual accepted syntax before deploying — this exact substitution mistake would either throw a config parse error on broker startup (the safer failure) or, depending on the parser’s leniency, silently fall back to a default the operator didn’t intend.

7. Throughput tuning has a latency cost — the trade-off this material doesn’t state explicitly

Every throughput-improving change in §1–§3 works by making the client wait longer before acting, in exchange for doing more work per action. This is a real trade-off, not a pure win, and it’s worth stating explicitly rather than presenting “more throughput” as free.

linger.ms=20        → every record now waits up to 20ms longer before being sent
fetch.min.bytes=1MB → a consumer on a LOW-traffic topic can wait up to fetch.max.wait.ms
                       (500ms in §3's example) for that 1MB to accumulate, even if only
                       a handful of small messages are actually available

When this trade-off is wrong for a workload: the fetch.min.bytes: 1MB setting in §3 is genuinely risky for a topic that doesn’t produce anywhere near 1MB within fetch.max.wait.ms — instead of “faster consumption,” the practical effect can be “every fetch waits the full 500ms for data that will never accumulate to 1MB,” adding latency without any throughput benefit at all on a low-volume topic. These consumer-tuning settings are appropriate for genuinely high-throughput topics where 1MB accumulates quickly; applying them uniformly across every topic in a cluster, including low-traffic ones, actively hurts the low-traffic topics’ latency for no gain. Tune per-topic-characteristics, not as a single global consumer default.

8. `vm.dirty_ratio=80` — a durability trade-off, not just a throughput win

Allowing up to 80% of RAM to hold unflushed (“dirty”) page cache data before the OS forces a flush maximizes the benefit of relying on page cache (exactly as §4’s “rely on OS page cache flush” comment intends) — but it also means a much larger amount of data is vulnerable to loss in the event of a sudden power failure or kernel panic, before Kafka’s own replication (Day 10) even gets a chance to matter.

Why replication doesn’t fully cover for this: if a broker crashes with a large amount of unflushed dirty page cache and the OS-level buffer is lost (not gracefully flushed), that broker’s on-disk log segments can be left in an inconsistent state until Kafka’s own recovery/truncation logic reconciles them against the ISR on restart — this is exactly the class of problem acks=all + min.insync.replicas=2 (Day 10 §3) is designed to protect the cluster’s durability against, but it’s still worth understanding that a very high dirty_ratio increases the amount of work/data at risk on any single broker during an ungraceful crash, rather than treating the setting as a pure performance dial with no downside. A more conservative vm.dirty_ratio (many production Kafka deployments use values in the 10-40% range rather than 80%) trades some throughput for a smaller window of at-risk unflushed data per broker — validate this specific number against your actual power-loss/crash risk tolerance rather than adopting 80% by default.

9. Reconciling with Day 13 — zstd vs snappy

§1’s compression table marks snappy with a ★ as the balanced recommendation — but Day 13 §3 established zstd as the general-purpose default (“gzip-level compression ratio at near-lz4 speed”), and even this document’s own table (§1) shows zstd beating snappy on both speed and ratio.

The actual guidance, reconciled: zstd is the better default for most workloads in Kafka 4 — it was specifically called out in Day 13 §3 as the recommended choice since Kafka 2.1 introduced support for it, and nothing about this performance-tuning context changes that recommendation. snappy remains a reasonable choice specifically when compatibility with older Kafka client libraries that predate broad zstd support is a real constraint, or when an organization has existing tooling/benchmarks built around snappy — not as a generally “safer” default over zstd for a Kafka 4 deployment. If starting fresh (as this course’s labs-api/labs-socket project is), prefer zstd over snappy unless there’s a specific compatibility reason not to.

10. Benchmark against realistic topology, not a single dev broker

§1 and §3’s perf-test commands (kafka-producer-perf-test.sh/kafka-consumer-perf-test.sh) are shown pointing at a single broker:9092 — useful for a first pass, but results from a single-broker setup can meaningfully mislead about production behavior, since replication (Day 10), network topology (Day 51’s broker-count math), and realistic partition/consumer-group counts all change the numbers.

What to change for a trustworthy benchmark: run the same perf tests against a multi-broker cluster with the actual replication.factor/min.insync.replicas production settings applied (§1’s acks=all in particular changes throughput meaningfully compared to a default acks=1 single-broker test), the actual partition count from Day 51’s sizing exercise, and — where possible — production-representative network latency between client and cluster (a benchmark run from the same host as the broker understates real-world network overhead). A single-broker, same-host benchmark is a reasonable smoke test for “did this config change break anything obviously,” but shouldn’t be the basis for capacity-planning decisions that assume the tuned numbers will hold in production.

11. Common pitfalls

Copying Long.MAX_VALUE literally into server.properties — needs the actual numeric value (9223372036854775807), not the Java constant name (§6)
Applying high fetch.min.bytes uniformly across both high- and low-traffic topics — actively hurts latency on low-volume topics without any throughput benefit, since the wait for fetch.max.wait.ms rarely pays off (§7)
Setting vm.dirty_ratio aggressively high without weighing the durability trade-off — more unflushed data at risk per broker on an ungraceful crash, not a purely positive throughput setting (§8)
Defaulting to snappy over zstd without a specific compatibility reason — Day 13’s guidance and this very document’s own comparison table both point to zstd as the better general-purpose choice (§9)
Trusting single-broker, same-host benchmark numbers as production capacity planning input — misses the throughput impact of real replication (acks=all), realistic partition counts, and actual network latency (§10)

Key Takeaways

Raise batch.size (64KB) + linger.ms (20ms) for high-throughput producers — fewer, larger requests, at the cost of added per-record latency (§7)
Prefer zstd over snappy as the general default — matches Day 13’s guidance and this document’s own comparison numbers; reserve snappy for specific compatibility constraints
Consumer: set fetch.min.bytes/concurrency per actual topic throughput characteristics, not as a uniform global default — high fetch.min.bytes actively hurts low-traffic topics
Broker: fix JVM heap (-Xms = -Xmx = 6g), and treat vm.swappiness=1 and vm.dirty_ratio as durability-aware choices, not pure throughput wins
Copy config values from verified working files, not documentation prose — Long.MAX_VALUE as literal text is a real, easy mistake
Page cache is Kafka’s secret weapon — leave half the RAM for the OS page cache
Measure first, and measure realistically: use kafka-producer-perf-test.sh against a production-representative multi-broker topology with real acks/replication settings, not a single-broker same-host test

Support me through GitHub Sponsors.

Thank you for Reading !! See you in the next post.

➡️ Day 53: Confluent Cloud, MSK, Aiven — managed Kafka basics

Resources

📘 Kafka: The Definitive Guide — Chapters 4 (Producers) & 5 (Consumers)
🌐 kafka.apache.org/documentation/#producerconfigs
🌐 kafka.apache.org/documentation/#consumerconfigs

Link to Medium blog