60-Day Kafka 4 Learning Plan · Week 2 — Kafka 4 Internals Sources: Kafka: The Definitive Guide Ch.3 · kafka.apache.org/43/documentation/#producerconfigs
Goal
Understand how Kafka producer batch compression works, compare gzip, snappy, lz4 and zstd, understand the consumer- and broker-side costs of compression choices, and configure compression correctly in a Spring Boot 3 producer.
1. Where compression happens
Producer batch → Compress batch → Send over wire → Broker stores as-is
(N records, (compression. (smaller (compressed on
raw bytes) type) payload) disk too)Compression applies to the whole producer batch, not individual messages. Bigger batches compress noticeably better than small ones.
The broker does not decompress the batch — it stores and forwards it compressed as-is (unless a format conversion is required, e.g. for old consumers). The consumer decompresses on read.
This saves both network bandwidth and disk space end-to-end.
2. Codec comparison
3. Which codec should you pick?
Default choice: zstd ★
Best ratio + speed balance for most applications. Default recommendation since Kafka 2.1 introduced zstd support.
Lowest latency: lz4
Use when CPU is the bottleneck, not network — e.g. high-frequency trading systems where every millisecond counts.
Bandwidth-critical: gzip
Cross-region replication or metered network links where bandwidth cost outweighs CPU cost.
Avoid: none
Only acceptable for tiny payloads where compression overhead exceeds the gain.
4. Batch size matters more than the codec
Compression works on the whole batch — bigger batches mean better compression ratios for any codec.
Tune batch.size and linger.ms together — give the producer time to accumulate a worthwhile batch before sending.
5. Consumer-side decompression cost
Compression isn’t free on the read side either — every consumer that reads a batch pays the decompression cost, and that cost is paid once per consumer group, not once per message.
1 producer compresses a batch once
→ N consumer groups each independently decompress that same batch on read
→ decompression CPU cost scales with (batch size × number of consumer groups reading it)Practical implications:
- A topic read by many downstream consumer groups (fan-out architectures, CDC topics feeding multiple services) multiplies decompression cost — a heavier codec like
gzipbecomes more expensive in aggregate than the same choice on a single-consumer topic. lz4andzstddecompress significantly faster thangzipfor the same data, which matters more as fan-out increases — this is a second reason (beyond producer CPU) to prefer them overgzipfor high-fan-out topics.- Consumer lag investigations should account for decompression time on the consumer’s critical path, especially for consumers on constrained CPU (e.g. small containers) reading
gzip-compressed batches.
6. Spring Boot producer config
spring:
kafka:
producer:
compression-type: zstd
batch-size: 65536 # 64 KB — bigger batch, better compression ratio
properties:
linger.ms: 20 # wait up to 20ms to fill the batch
buffer.memory: 33554432 # 32 MB total producer buffer
# zstd compression level (default 3, range 1-22, higher = better ratio, slower)
compression.zstd.level: 3Programmatic config
@Bean
public ProducerFactory<String, String> producerFactory() {
Map<String, Object> config = new HashMap<>();
config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
config.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "zstd");
config.put(ProducerConfig.BATCH_SIZE_CONFIG, 65536);
config.put(ProducerConfig.LINGER_MS_CONFIG, 20);
return new DefaultKafkaProducerFactory<>(config);
}7. Topic-level vs producer-level config — and the recompression cost
Topic configuration can override the producer’s compression choice.
# Force a specific codec for a topic, regardless of producer setting
kafka-configs.sh --bootstrap-server localhost:9092 \
--entity-type topics --entity-name orders --alter \
--add-config compression.type=zstd
# Or inherit whatever the producer sends (default behaviour)
kafka-configs.sh --bootstrap-server localhost:9092 \
--entity-type topics --entity-name orders --alter \
--add-config compression.type=producer
Hidden cost: when a topic’s
compression.typediffers from what the producer sends, the broker must decompress the incoming batch and recompress it with the configured codec — on every single write, on every leader broker for that partition. This is real, sustained CPU cost on your brokers, not a one-time migration cost. Prefer aligning producer and topic compression settings; only use a topic-level override when you genuinely can’t control every producer writing to that topic (e.g. shared platform topics with many teams as clients).
8. Monitoring compression effectiveness
Diagnosis tip: if
compression-rate-avglooks worse than the benchmark table in §9 for the same codec, suspect the payload shape before the codec choice — binary blobs, encrypted fields, or already-compressed attachments (images, video, gzip-in-gzip) compress poorly regardless of algorithm.
9. Real-world JSON payload benchmark*
*Approximate figures from typical text/JSON payloads. Always benchmark with your actual data shape — binary, highly-structured, or already-compressed payloads (images, video) will behave very differently.
Benchmarking with your own data
Don’t trust generic tables for production sizing decisions — run a quick local comparison against representative production-shaped payloads:
byte[] sample = loadRepresentativeSampleBatch(); // real production payload shape, not synthetic data
for (CompressionType codec : List.of(GZIP, SNAPPY, LZ4, ZSTD)) {
long start = System.nanoTime();
byte[] compressed = compress(sample, codec);
long elapsedMs = (System.nanoTime() - start) / 1_000_000;
System.out.printf("%s: %.1fx ratio, %dms%n",
codec, (double) sample.length / compressed.length, elapsedMs);
}Schema changes (new fields, different field ordering, switching from JSON to Avro/Protobuf) can meaningfully shift the ratio you actually get in production — re-benchmark after significant payload shape changes, not just once at initial rollout.
10. Common pitfalls
- Choosing a codec from a generic benchmark table without testing real payloads — already-compressed fields (e.g. a base64-encoded image inside a JSON payload) can make any codec look far worse than expected
- Small
batch.size+ expensive codec — pays the CPU cost ofgzipwithout getting its compression-ratio benefit, since there’s not enough data per batch to compress well - Topic-level
compression.typeoverride left in place indefinitely — silently taxes broker CPU on every write; revisit whether producers can just send the desired codec directly instead - Ignoring consumer-side cost in high-fan-out topics — a codec choice optimized purely for producer/network cost can become a bottleneck once multiplied across many consumer groups
- Not re-benchmarking after schema/serialization format changes — a codec tuned for a JSON payload may behave very differently once the team migrates to Avro or Protobuf
Key Takeaways
- zstd is the best default — gzip-level compression ratio at near-lz4 speed
- Compression applies to the whole producer batch — bigger batch = better ratio
- Tune
batch.size+linger.mstogether to let bigger batches form before sending - Broker stores compressed batches as-is — saves disk AND network bandwidth
- lz4 wins when CPU is the bottleneck; gzip wins when network cost is the bottleneck
- Decompression cost is paid by every consumer group independently — high-fan-out topics amplify the cost of a heavy codec like gzip
- Topic-level
compression.typeoverrides force the broker to decompress and recompress on every write — real, sustained CPU cost, not a one-time migration cost - Benchmark with representative production payloads, not generic tables — and re-benchmark after schema/serialization changes
Support me through GitHub Sponsors.
Next
➡️ Day 14: Internal lab — inspect KRaft logs, tune replication
Resources
- 📘 Kafka: The Definitive Guide — Chapter 3 (Kafka Producers — compression)
- 🌐 kafka.apache.org/43/documentation/#producerconfigs —
compression.type
