60-Day Kafka 4 Learning Plan · Week 3 — Day 16 of 60

60-Day Kafka 4 Learning Plan · Week 3 — Spring Boot Integration Sources: Kafka: The Definitive Guide Ch.3 · kafka.apache.org/43/documentation/#producerconfigs

Goal

Understand the three acks levels, how retries and delivery timeout interact, what idempotent producer mode does, how ordering is preserved under retries, and how to wire it all together — with monitoring and tests — in a Spring Boot 3 application.

1. acks — acknowledgement levels

acks=0 (fire & forget)

No acknowledgement waited for. Highest possible throughput. Data loss is possible — the broker may not have even received the message. Use for: metrics, non-critical log events.

acks=1 (leader ack)

The leader broker writes the record and acknowledges. Good throughput. Data loss possible if the leader fails before replicating to followers. Use for: general-purpose messaging where some loss is acceptable.

acks=all (-1) ★

All in-sync replicas (ISR) must acknowledge before the broker responds. No data loss when combined with min.insync.replicas=2. Higher latency. Use for: orders, payments, financial events — anything where data loss is unacceptable.

2. Retries & delivery timeout

Kafka retries transient errors automatically — network blips, leader elections, broker restarts.

delivery.timeout.ms is the master clock. Retries stop when it expires, regardless of the retries count. Set it to cover your acceptable outage window.

Not every failure is retriable. Transient errors (TimeoutException, NotLeaderForPartitionException) get retried automatically; fatal ones (RecordTooLargeException, SerializationException, AuthorizationException) fail immediately regardless of retries — see the retriable-vs-fatal breakdown from Day 5 §6 for the full picture and dead-letter handling patterns.

3. Idempotent producer — exactly-once writes

The problem without idempotence

Producer sends message
    → Network drops the ack
    → Producer retries
    → Broker stores the message TWICE
    → Consumer processes the same event twice ⚠

The solution: `enable.idempotence=true`

Each producer is assigned a unique producer_id. Every batch is tagged with a monotonically increasing sequence number per partition. The broker deduplicates any retry with the same (producer_id, sequence_number) pair — the duplicate is silently discarded.

Result: exactly-once write semantics per partition, even under retries.

What idempotence auto-sets

When you set enable.idempotence=true, Kafka automatically configures:

acks=all — required for deduplication to work
retries=Integer.MAX_VALUE — retries indefinitely within delivery.timeout.ms
max.in.flight.requests.per.connection=5 — Kafka 4 maintains ordering with up to 5 in-flight batches

You only need to set enable.idempotence=true — the rest follows automatically.

4. Ordering guarantees under retries — why `max.in.flight=5` is safe

Without idempotence, allowing multiple in-flight requests per connection is dangerous: if batch 2 succeeds but batch 1 fails and retries, batch 2’s data can land on the broker before batch 1’s retry — silently reordering records within the partition.

WITHOUT idempotence, max.in.flight=5:
  Batch 1 sent → fails, will retry
  Batch 2 sent → succeeds immediately
  Batch 1 retry → succeeds
  Broker order: [Batch 2, Batch 1]  ⚠ REORDERED

How idempotence fixes this: each batch carries a sequence number, and the broker enforces strict sequence ordering per partition — it will reject (and the client will internally retry/reorder) any batch that arrives out of sequence, even across multiple in-flight requests.

WITH idempotence, max.in.flight=5:
  Batch 1 (seq=10) sent → fails, will retry
  Batch 2 (seq=11) sent → broker holds it, seq=10 hasn't landed yet
  Batch 1 retry (seq=10) → accepted
  Batch 2 (seq=11) → now accepted, correct order preserved

Practical rule: never set max.in.flight.requests.per.connection > 5 even with idempotence enabled — that’s the maximum the broker’s in-order sequence tracking supports (OutOfOrderSequenceException otherwise). And never raise it above 1 without idempotence if ordering matters at all.

5. min.insync.replicas — the other half of acks=all

acks=all alone only guarantees the write reaches whatever the current ISR is — it says nothing about how large that ISR needs to be. min.insync.replicas is what turns acks=all into an actual durability guarantee rather than a formality.

replication.factor=3, min.insync.replicas=2, acks=all

ISR = [leader, follower-A, follower-B]        → write succeeds, 3 copies
ISR = [leader, follower-A]  (B fell behind)   → write succeeds, 2 copies — still safe
ISR = [leader]              (A and B down)    → write REJECTED — NotEnoughReplicasException

Without min.insync.replicas >= 2, an ISR that’s shrunk to just the leader still satisfies acks=all — because “all ISR members acked” is trivially true when there’s only one member. This is why the Day 10 golden rule (replication.factor=3 + min.insync.replicas=2 + acks=all) is a package deal, not three independent settings.

kafkaTemplate.send("orders", key, payload)
    .whenComplete((result, ex) -> {
        if (ex instanceof NotEnoughReplicasException) {
            // safe to retry — idempotence prevents duplicates once ISR recovers
            log.warn("Insufficient replicas, will retry", ex);
        }
    });

6. Monitoring producer reliability

# Confirm current ISR state when investigating NotEnoughReplicasException spikes
kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders --under-min-isr-partitions

Correlate, don’t guess: a spike in record-retry-rate on the producer side paired with UnderMinIsrPartitionCount > 0 on the broker side (Day 10 §11) tells you the retries are a symptom of ISR trouble, not a producer-side bug — check broker health before touching client config.

7. Testing acks/idempotence behavior

class OrderProducerRetryTest {

    @Test
    void deduplicatesRetriedRecordWithIdempotence() {
        MockProducer<String, String> mockProducer =
            new MockProducer<>(true, new StringSerializer(), new StringSerializer());
        // Note: MockProducer doesn't fully simulate broker-side sequence dedup —
        // use @EmbeddedKafka for a true end-to-end idempotence test.
        KafkaTemplate<String, String> template = new KafkaTemplate<>(new MockProducerFactory<>(mockProducer));

        template.send("orders", "o1", "payload");
        assertThat(mockProducer.history()).hasSize(1);
    }
}

@SpringBootTest
@EmbeddedKafka(partitions = 1, topics = "orders")
class IdempotentProducerIntegrationTest {

    @Autowired KafkaTemplate<String, String> kafkaTemplate;

    @Test
    void singleSendProducesExactlyOneRecord() throws Exception {
        kafkaTemplate.send("orders", "o1", "payload").get(5, TimeUnit.SECONDS);
        // consume and assert exactly one record — true dedup behavior only
        // observable against a real (or embedded) broker enforcing sequence numbers
    }
}

Why MockProducer isn’t enough here: it records what your application sent, not what the broker deduplicated — testing actual idempotent dedup behavior requires @EmbeddedKafka (or a real cluster), since the sequence-number enforcement lives on the broker, not the client.

8. Common pitfalls

Setting retries low (e.g. retries: 3) alongside enable.idempotence=true — idempotence auto-sets retries=MAX, but an explicit low value in application.yml can override that; if durability matters, let delivery.timeout.ms be the real bound instead of capping retries manually
acks=all without min.insync.replicas >= 2 — durability theater, as covered in §5; the write “succeeds” against an ISR of just the leader
Raising max.in.flight.requests.per.connection above 5 — breaks the broker’s in-order sequence tracking guarantee, even with idempotence on
Assuming idempotence covers multi-partition atomicity — it only dedups within a single partition/producer session; cross-partition atomicity needs transactions (Day 11)
Ignoring record-retry-rate until errors appear — retries are the leading indicator; by the time record-error-rate moves, delivery.timeout.ms may already be close to exhausted for in-flight batches

9. Spring Boot producer config

# application.yml — production-grade producer
spring:
  kafka:
    producer:
      acks: all                  # require all ISR to ack
      retries: 3                 # explicit limit (delivery.timeout is the real guard)
      properties:
        enable.idempotence: true         # no duplicates on retry
        delivery.timeout.ms: 120000      # 2 min total time budget
        request.timeout.ms: 30000        # 30 s per individual attempt
        retry.backoff.ms: 500            # 500 ms between retries
        max.in.flight.requests.per.connection: 5
        # max.in.flight=5 is safe with idempotence=true in Kafka 4

Programmatic equivalent

@Bean
public ProducerFactory<String, String> producerFactory() {
    Map<String, Object> config = new HashMap<>();
    config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
    config.put(ProducerConfig.ACKS_CONFIG, "all");
    config.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
    config.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 120_000);
    config.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, 30_000);
    config.put(ProducerConfig.RETRY_BACKOFF_MS_CONFIG, 500);
    config.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);
    return new DefaultKafkaProducerFactory<>(config);
}

10. Quick decision guide

Key Takeaways

acks=all + min.insync.replicas=2 = no data loss with up to 1 broker failure — the two settings are a package deal, not independent knobs
delivery.timeout.ms is the master clock — retries stop when it expires
enable.idempotence=true deduplicates retries using producer_id + sequence number
Idempotence auto-sets acks=all and retries=MAX — but an explicit low retries value in config can silently override that
max.in.flight.requests.per.connection=5 is the safe ceiling with idempotence — the broker enforces in-order sequence acceptance up to that limit, never higher
record-retry-rate is the leading indicator for producer reliability problems — watch it before record-error-rate spikes
MockProducer tests application-level send calls; true idempotent dedup behavior needs @EmbeddedKafka since deduplication happens broker-side
For transactional exactly-once across topics, use KafkaTransactionManager (Day 18)

Support me through GitHub Sponsors.

➡️ Day 17: Consumer config — concurrency, offsets, batch listeners

Resources

📘 Kafka: The Definitive Guide — Chapter 3 (Kafka Producers)
🌐 kafka.apache.org/43/documentation/#producerconfigs

👉 Link to Medium blog