60-Day Kafka 4 Learning Plan · Week 2 — Day 12 of 60

---

60-Day Kafka 4 Learning Plan · Week 2 — Kafka 4 Internals Sources: Kafka: The Definitive Guide Ch.3 · kafka.apache.org/43/documentation/#producerconfigs

Goal

Understand how Kafka uses message keys to determine partition assignment, the difference between the default murmur2 hash and the sticky partitioner, how to implement a custom partitioner, how to detect and mitigate hot partitions, and how co-partitioning enables joins in Kafka Streams.

1. Why keys matter

All messages with the same key always land on the same partition.

Partition is the unit of ordering in Kafka. Same partition = strict ordering guaranteed.

If ordering doesn’t matter, use null key for maximum throughput.

2. Default partitioner — MurmurHash2

When a key is provided, Kafka computes:

partition = abs(murmur2(keyBytes)) % numPartitions

key="U42"  →  murmur2("U42") = 1234567890  →  1234567890 % 6 = 0  →  Partition 0

Properties:

• Deterministic — same key always maps to the same partition
• Good statistical distribution across partitions
• Fast (non-cryptographic hash)

⚠️ Warning: if you increase numPartitions after go-live, existing keys will map to different partitions. Plan partition count upfront.

3. No key — Sticky Partitioner

When key=null, Kafka 4 uses the Sticky Partitioner (default since Kafka 2.4).

The sticky partitioner drastically reduces the number of requests sent to brokers by coalescing records into larger batches before switching partitions.

4. Partitioning strategies comparison

5. Custom partitioner — VIP routing example

Route VIP orders to partition 0 exclusively; hash all others normally.

public class VipPartitioner implements Partitioner {

    @Override
    public int partition(String topic, Object key, byte[] keyBytes,
                         Object value, byte[] valueBytes, Cluster cluster) {
        int numPartitions = cluster.partitionCountForTopic(topic);

        // VIP orders always go to partition 0
        if (key != null && key.toString().startsWith("VIP-")) {
            return 0;
        }

        // All others: default murmur2 hash
        return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions;
    }

    @Override
    public void configure(Map<String, ?> configs) {}

    @Override
    public void close() {}
}

Note: In Kafka 4, you can also implement the newer Partitioner API via ProducerInterceptor or use the partitioner.class property (still supported).

6. Register custom partitioner in Spring Boot

# application.yml
spring:
  kafka:
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer
      properties:
        partitioner.class: com.boottech.kafka.VipPartitioner

Or programmatically:

@Bean
public ProducerFactory<String, String> producerFactory() {
    Map<String, Object> config = new HashMap<>();
    config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
    config.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
    config.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
    config.put(ProducerConfig.PARTITIONER_CLASS_CONFIG, VipPartitioner.class);
    return new DefaultKafkaProducerFactory<>(config);
}

7. Explicit partition assignment

Sometimes the producer should bypass hashing entirely and target a specific partition directly — useful for admin tooling, backfills, or reprocessing a known-bad partition.

// KafkaTemplate overload that takes an explicit partition
kafkaTemplate.send("orders", 2, "VIP-001", payload); // force partition 2, key still stored for ordering/compaction

// Raw ProducerRecord form
ProducerRecord<String, String> record =
    new ProducerRecord<>("orders", 2, "VIP-001", payload); // (topic, partition, key, value)
producer.send(record);

Careful: explicit partition assignment bypasses the key-hash guarantee entirely. If other producers are writing the same logical key via normal hashed sends, you can end up with the same key’s messages split across two partitions — breaking the ordering guarantee you were relying on. Only use this when you control all writers for that key.

8. Co-partitioning — a prerequisite for Kafka Streams joins

Co-partitioning means two topics have the same number of partitions and use the same key for the same logical entity — so a join between them can be done locally, partition-by-partition, without a network shuffle.

orders topic   (6 partitions, key=userId)
payments topic (6 partitions, key=userId)

Partition 3 of orders   ←── userId hashes here ──┐
Partition 3 of payments ←── same userId, same hash ─┘  → can join locally, no shuffle

If the two topics have different partition counts, hash(key) % numPartitions produces different results per topic — the same userId could land on partition 2 of one topic and partition 5 of the other, making a local join impossible. Kafka Streams will throw a TopologyException at startup if it detects non-co-partitioned inputs to a join.

Practical rule: any topics you plan to join in Kafka Streams (covered in a later week) must be created with identical partition counts from day one — this is one more reason to plan partition count upfront rather than resizing later.

9. Monitoring partition balance & hot partitions

# Check per-partition disk usage — a fast way to spot a skewed hot partition
kafka-log-dirs.sh --bootstrap-server localhost:9092 --topic-list orders --describe

# Compare partition sizes directly from broker log dirs
du -sh /var/kafka-logs/orders-*/

Diagnosis tip: if du shows one partition directory significantly larger than its siblings while message counts look similar, suspect a hot key rather than a broker-level issue — check application-level key distribution before touching cluster config.

10. Testing partitioner logic

class VipPartitionerTest {

    private final VipPartitioner partitioner = new VipPartitioner();
    private final Cluster cluster = someTestClusterWith(6, "orders"); // helper: 6 partitions

    @Test
    void routesVipKeysToPartitionZero() {
        int partition = partitioner.partition("orders", "VIP-001", "VIP-001".getBytes(),
            null, null, cluster);
        assertThat(partition).isEqualTo(0);
    }

    @Test
    void routesRegularKeysConsistently() {
        int p1 = partitioner.partition("orders", "user-42", "user-42".getBytes(), null, null, cluster);
        int p2 = partitioner.partition("orders", "user-42", "user-42".getBytes(), null, null, cluster);
        assertThat(p1).isEqualTo(p2); // deterministic — same key, same partition every time
    }
}

Test determinism explicitly (same key → same partition across calls) — it’s the property your whole ordering guarantee depends on, and it’s easy to accidentally break with a partitioner that has hidden state or relies on wall-clock time.

11. Hot partition problem & mitigation

Hot partition: one key carries the majority of traffic → one partition gets all the load.

Symptoms:

• One broker CPU/disk consistently high
• Consumer lag growing only on specific partitions
• Replication pressure on the leader replica

Mitigation strategies

1. Key salting — append a random suffix to distribute traffic:

// Instead of key = "VIP-001"
String key = "VIP-001-" + ThreadLocalRandom.current().nextInt(0, 10);
// Consumer aggregates results from all 10 sub-keys

2. Custom partitioner — spread VIP keys across multiple dedicated partitions:

if (key.startsWith("VIP-")) {
    return vipPartitionIndex++ % NUM_VIP_PARTITIONS;
}

3. Increase partition count — gives more parallelism, but breaks existing key mapping.

4. Dedicated VIP topic — separate orders-vip topic with its own consumer group and scaling policy.

Salting trade-off to plan for: once you salt a key, per-entity ordering across the salted sub-keys is gone — a consumer that needs the true global order for VIP-001 events must now reassemble ordering itself (e.g. via an event timestamp) instead of relying on partition ordering. Only salt keys where you can tolerate reordering within that key or can reconstruct it downstream.

12. Common pitfalls

• Changing partition count on a live topic without a migration plan — existing keys silently remap, breaking assumed ordering for anything already in flight or relying on hash(key) % N staying stable
• Mixing explicit-partition sends with hashed sends for the same key — splits a logical entity’s events across partitions, silently breaking ordering
• Joining two Kafka Streams topics with different partition counts — fails at startup with TopologyException; must be caught at topic-creation time, not discovered in production
• Salting hot keys without a reassembly strategy — solves the throughput problem but silently discards per-key ordering unless the consumer explicitly reconstructs it
• Assuming a custom partitioner is free — every send now runs your routing logic; keep it allocation-light (the VIP example does a string prefix check + hash, which is cheap — avoid regexes or lookups in the hot path)

Key Takeaways

• Same key → same partition always — this is how Kafka guarantees per-key ordering
• Default: MurmurHash2(key) % numPartitions — deterministic, good distribution
• Null key → Sticky Partitioner (default) — fills one batch before rotating, better throughput
• Implement Partitioner interface for custom routing (VIP, tenant, geography)
• Adding partitions changes the key→partition mapping — plan partition count upfront, especially for topics that will be joined in Kafka Streams (co-partitioning requires matching partition counts)
• Explicit partition assignment bypasses key hashing — only safe when you control all writers for that key
• Hot partitions: add key salt or use a custom partitioner to spread load evenly, but salting trades away per-key ordering
• Monitor per-partition BytesInPerSec and use kafka-log-dirs.sh to catch skew before it becomes an incident

---

Support me through GitHub Sponsors.

Next

➡️ Day 13: Compression — gzip, snappy, lz4, zstd tradeoffs

Resources

• 📘 Kafka: The Definitive Guide — Chapter 3 (Kafka Producers)
• 🌐 kafka.apache.org/43/documentation/#producerconfigs — partitioner.class
• 🌐 KIP-794: Sticky Partitioner improvements

👉 Link to Medium blog