60-Day Kafka 4 Learning Plan · Week 2 — Day 9 of 60

---

60-Day Kafka 4 Learning Plan · Week 2 — Kafka Internals Sources: Kafka: The Definitive Guide Ch.5 · kafka.apache.org/43

Goal

Understand how Kafka stores data on disk: partition directory layout, the three file types per segment, when segments roll, how delete and compact retention policies work, tiered storage, and how to configure and monitor retention from Spring Boot and the broker.

1. Partition directory on disk

A partition is a directory on the broker filesystem. Inside it lives an ordered sequence of segments.

orders-0/                          ← partition directory
  00000000000000000000.log         ← closed segment (offsets 0–499)
  00000000000000000000.index
  00000000000000000000.timeindex
  00000000000000000500.log         ← closed segment (offsets 500–999)
  00000000000000000500.index
  00000000000000000500.timeindex
  00000000000000001000.log         ← ACTIVE segment (offsets 1000–now)
  00000000000000001000.index
  00000000000000001000.timeindex

Key rules:

• The filename is the base offset of the first message in that segment.
• There is always exactly one active segment — the one receiving new writes.
• Closed segments are eligible for deletion/compaction; the active segment is never deleted.

2. Three files per segment

.log — raw message bytes

• Sequential, append-only binary file.
• Each record contains: offset, timestamp, key bytes, value bytes, headers.
• This is the source of truth; everything else indexes into it.

.index — sparse offset index

• Maps relative offset → byte position in the .log file.
• Not every offset is indexed — sampled every log.index.interval.bytes (default 4096 bytes).
• Loaded entirely into memory on startup for fast binary search O(log n).

.timeindex — sparse time index

• Maps timestamp → offset in the .index file.
• Enables consumer.offsetsForTimes(timestamp) — seek to a point in time.
• Also sparse, same sampling interval.

Read path for a consumer seeking offset N:

.timeindex (if time seek) → .index (offset → byte pos) → .log (read bytes)

3. Segment rolling

A new active segment is created when any of these triggers fires:

When a segment rolls: the current .log / .index / .timeindex are sealed (closed), and a new triple of files is created with the next offset as the filename.

4. Retention policies

Delete policy (default)

log.cleanup.policy=delete

Kafka deletes whole closed segments once they exceed the retention threshold. The active segment is never deleted regardless.

Time-based:

log.retention.hours=168          # 7 days (default)
log.retention.ms=86400000        # 1 day (overrides hours if set)

Size-based:

log.retention.bytes=-1           # disabled by default
                                 # per-partition cap when set

Whichever threshold triggers first wins. Both can be active simultaneously.

Compact policy

log.cleanup.policy=compact

Instead of deleting entire segments, Kafka runs a log cleaner that:

1. Scans segments for duplicate keys.
2. Keeps only the latest record per key.
3. Removes records whose key has a tombstone (null value) — the tombstone itself is also eventually removed.

Good for: changelogs, event-sourced aggregates, materialised views (Kafka Streams).

Tune aggressiveness:

min.cleanable.dirty.ratio=0.5    # compact when 50% of log is "dirty" (default)

Combine both

log.cleanup.policy=delete,compact

Compacts first (deduplicate), then deletes segments older than the retention window.

5. Log compaction — before & after

BEFORE compaction:                AFTER compaction:
────────────────────              ────────────────────────
offset 0 · key=A · val=1   ──┐   offset 2 · key=A · val=2  ✓ latest
offset 1 · key=B · val=x   ──┼─  offset 3 · key=C · val=y  ✓ kept
offset 2 · key=A · val=2   ←─┘   (key=B deleted — tombstone at offset 4)
offset 3 · key=C · val=y
offset 4 · key=B · null  ← tombstone

6. Compaction internals — the log cleaner

Compaction isn’t instant or synchronous with writes — it’s done by background log cleaner threads that periodically scan eligible segments.

log.cleaner.threads=1                    # background cleaner threads per broker
log.cleaner.min.compaction.lag.ms=0      # min time a message stays uncompacted (protects recent writes)
log.cleaner.max.compaction.lag.ms=Long.MAX_VALUE  # force compaction after this age, even if dirty ratio not hit
delete.retention.ms=86400000             # how long a TOMBSTONE is kept before physical removal (default 24h)

Why delete.retention.ms matters: if a tombstone is removed too quickly, a consumer that’s behind (still reading older offsets) might read the old value for a key without ever seeing the deletion — because the “this key was deleted” signal is gone. Keeping tombstones around for 24h+ gives slow consumers time to observe the deletion before it disappears.

Dirty ratio calculation:

dirty ratio = bytes in "dirty" (uncompacted) segments / total bytes in the partition

Once this crosses min.cleanable.dirty.ratio (default 0.5), the segment becomes eligible for the next cleaner pass — it doesn’t compact instantly, so a compacted topic can briefly contain duplicate keys until the next cleaning cycle runs.

Gotcha: the active segment is never compacted — only closed segments. A key updated frequently in the active segment will show duplicates until that segment rolls and becomes eligible.

7. Tiered storage (KIP-405) — Kafka 4

Traditionally, retention was bounded by local broker disk. Tiered storage offloads older closed segments to cheaper remote storage (S3, HDFS, GCS) while keeping recent data on fast local disk — decoupling retention length from local disk cost.

# Broker-level
remote.log.storage.system.enable=true

# Per-topic
remote.storage.enable=true
local.retention.ms=3600000        # keep only 1h locally — fast reads for recent data
retention.ms=2592000000           # 30 days total — rest served from remote tier

Local disk (fast, expensive):     [last 1h — recent segments]
Remote tier (slow, cheap):        [1h – 30 days — older segments, fetched on demand]

Read path change: consumers seeking recent offsets hit local disk as normal; consumers seeking far back transparently trigger a remote fetch — slower, but keeps years of history queryable without provisioning broker disks for it.

Use case fit: compliance/audit topics needing long retention, event-sourcing systems replaying history, or cost optimization on topics with rare-but-important historical reads.

8. Monitoring segments & retention

# Inspect segment file contents directly for debugging
kafka-dump-log.sh --files /var/kafka-logs/orders-0/00000000000000001000.log --print-data-log

9. Disk space planning

Rough sizing formula per partition, before replication:

partition size ≈ avg_message_size × messages_per_day × retention_days

Multiply by replication.factor for total cluster disk footprint, since every replica stores a full copy. For a topic with acks=all and replication.factor=3, a 100 GB logical dataset consumes 300 GB of raw disk across the cluster.

Compacted topics are the exception — their disk usage is bounded by unique key count, not by message volume over time, since old values for the same key get removed. This makes compacted changelog topics far cheaper to retain “forever” than delete-policy topics with the same throughput.

10. Common pitfalls

• Setting log.retention.bytes without log.retention.hours intending size-only retention — remember both thresholds are evaluated, and the first one hit wins; the default time-based 7-day retention is still active unless explicitly disabled
• Assuming compaction is synchronous — a just-published duplicate key can coexist with older versions until the next cleaner pass runs
• Setting delete.retention.ms too low on compacted topics — slow consumers may miss tombstones entirely and retain deleted keys indefinitely in their local state
• Ignoring uncleanable-partitions-count — a stuck cleaner silently stops compacting, and disk usage creeps up unnoticed until it’s a capacity emergency
• Using tiered storage without accounting for remote fetch latency — consumers replaying old history will see materially higher read latency than local-disk reads; not suitable for latency-sensitive backfills

11. Spring Boot: create topic with custom retention

@Configuration
public class KafkaTopicConfig {

    @Bean
    public NewTopic ordersTopic() {
        return TopicBuilder.name("orders")
                .partitions(3)
                .replicas(1)
                .config(TopicConfig.RETENTION_MS_CONFIG, "86400000")   // 1 day
                .config(TopicConfig.CLEANUP_POLICY_CONFIG, "compact")
                .build();
    }

    @Bean
    public NewTopic eventsTopic() {
        return TopicBuilder.name("events")
                .partitions(6)
                .replicas(1)
                .config(TopicConfig.RETENTION_MS_CONFIG, "604800000")  // 7 days
                .config(TopicConfig.SEGMENT_BYTES_CONFIG, "536870912") // 512 MB
                .build();
    }
}

Verify retention on a running topic:

kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --topic orders

12. Key configuration reference

All can be set at broker level (server.properties / KRaft config) or per-topic (overrides broker defaults).

Key Takeaways

• A partition = ordered list of closed segments + 1 active segment (always open, never deleted)
• Each segment has .log (data) + .index (offset lookup) + .timeindex (time-based lookup)
• Segments roll when any of: size, age, or index-size thresholds is hit
• Delete policy removes whole closed segments past retention; compact policy keeps latest value per key — but compaction runs asynchronously via background cleaner threads, not instantly
• Tombstones need a retention window (delete.retention.ms) so slow consumers don’t miss deletions
• Tiered storage (KIP-405) decouples long retention from local disk cost by offloading older segments to remote storage
• Disk planning: avg_message_size × messages_per_day × retention_days × replication_factor — except compacted topics, which scale with unique key count instead
• Spring Boot’s TopicBuilder lets you set per-topic retention overrides via TopicConfig.* constants

---

Support me through GitHub Sponsors.

Next

➡️ Day 10: Replication & Leader Election

Resources

• 📘 Kafka: The Definitive Guide — Chapter 5 (Kafka Internals)
• 🌐 kafka.apache.org/43/documentation/#brokerconfigs — log.* properties
• 🌐 Spring Kafka — TopicBuilder
• 🌐 KIP-405: Kafka Tiered Storage

👉 Link to Medium blog