60-Day Kafka 4 Learning Plan · Week 3 — Day 18 of 60

60-Day Kafka 4 Learning Plan · Week 3 — Spring Boot Integration Sources: Kafka: The Definitive Guide Ch.5 · docs.spring.io/spring-kafka/reference

Goal

Build a production-grade error handling pipeline in Spring Kafka: retry with exponential backoff (blocking and non-blocking), route unrecoverable messages to a dead-letter topic (DLT), distinguish retryable from non-retryable exceptions, and monitor/test the whole pipeline.

1. Error flow — retry then route to DLT

Message arrives
    → @KafkaListener throws exception
    → DefaultErrorHandler retries N times with exponential backoff
    → All retries exhausted
    → DeadLetterPublishingRecoverer publishes to {topic}.DLT

DLT naming convention: orders → orders.DLT

DLT message headers (set automatically by DeadLetterPublishingRecoverer):

kafka_dlt-original-topic — source topic name
kafka_dlt-original-partition — source partition
kafka_dlt-original-offset — source offset
kafka_dlt-exception-fqcn — fully qualified exception class name
kafka_dlt-exception-message — exception message

2. DefaultErrorHandler with exponential backoff

@Bean
public DefaultErrorHandler errorHandler(KafkaTemplate<?, ?> kafkaTemplate) {

    // 1. Recoverer: publish failed record to {topic}.DLT
    var recoverer = new DeadLetterPublishingRecoverer(kafkaTemplate,
        (record, exception) -> new TopicPartition(
            record.topic() + ".DLT",
            record.partition()
        )
    );

    // 2. Exponential backoff: 1s → 2s → 4s (max 3 retries)
    var backoff = new ExponentialBackOffWithMaxRetries(3);
    backoff.setInitialInterval(1_000L);   // 1 second
    backoff.setMultiplier(2.0);           // double each time
    backoff.setMaxInterval(10_000L);      // cap at 10 seconds

    return new DefaultErrorHandler(recoverer, backoff);
}

Backoff schedule: attempt 1 (immediate) → wait 1s → attempt 2 → wait 2s → attempt 3 → wait 4s → attempt 4 → exhausted → DLT.

This is blocking retry — the consumer thread sits idle waiting out each backoff interval before the next attempt, holding up that partition’s processing the entire time. §6 covers the non-blocking alternative for when that trade-off isn’t acceptable.

3. Retryable vs non-retryable exceptions

Not all exceptions should be retried — some will never succeed no matter how many times they’re attempted.

Retryable (transient — worth retrying)

TransientDataAccessException — temporary DB issue
OptimisticLockingFailureException — concurrent update conflict
ResourceAccessException — network timeout to a downstream HTTP service
RecoverableDataAccessException

Non-retryable (fatal — skip straight to DLT)

DeserializationException — malformed payload will never parse
ValidationException — invalid business data won’t become valid on retry
NullPointerException — code bug, retrying won’t help

@Bean
public DefaultErrorHandler errorHandler(KafkaTemplate<?, ?> kafkaTemplate) {
    var recoverer = new DeadLetterPublishingRecoverer(kafkaTemplate,
        (r, ex) -> new TopicPartition(r.topic() + ".DLT", r.partition()));

    var backoff = new ExponentialBackOffWithMaxRetries(3);
    backoff.setInitialInterval(1_000L);
    backoff.setMultiplier(2.0);

    var handler = new DefaultErrorHandler(recoverer, backoff);

    // These go straight to DLT, no retries
    handler.addNotRetryableExceptions(
        DeserializationException.class,
        ValidationException.class
    );

    return handler;
}

4. Wire error handler into listener container factory

@Bean
public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory(
        ConsumerFactory<String, String> consumerFactory,
        DefaultErrorHandler errorHandler) {

    var factory = new ConcurrentKafkaListenerContainerFactory<String, String>();
    factory.setConsumerFactory(consumerFactory);
    factory.setCommonErrorHandler(errorHandler);   // attach the handler
    factory.setConcurrency(3);
    return factory;
}

setCommonErrorHandler replaces the older setErrorHandler / setBatchErrorHandler from Spring Kafka 2.x — one unified API for both record and batch listeners in Spring Kafka 3.x.

5. Observe the DLT

Always consume the DLT in production — a growing DLT means your consumer has a systematic problem.

@KafkaListener(topics = "orders.DLT", groupId = "orders-dlt-monitor")
public void handleDlt(
        ConsumerRecord<String, String> record,
        @Header("kafka_dlt-exception-fqcn") String exceptionClass,
        @Header("kafka_dlt-exception-message") String errorMessage,
        @Header("kafka_dlt-original-offset") long originalOffset) {

    log.error("DLT message: key={} exception={} message={} originalOffset={}",
        record.key(), exceptionClass, errorMessage, originalOffset);

    // Options:
    // 1. Alert on-call (PagerDuty, Slack)
    alertingService.notify(record, errorMessage);

    // 2. Store for later replay after bug fix
    dltRepository.save(toDltEntry(record));
}

Replay DLT messages after a bug fix

// Read from DLT and re-publish to original topic
@KafkaListener(topics = "orders.DLT", groupId = "orders-dlt-replayer")
public void replayDlt(ConsumerRecord<String, String> record) {
    String originalTopic = new String(
        record.headers().lastHeader("kafka_dlt-original-topic").value()
    );
    kafkaTemplate.send(originalTopic, record.key(), record.value());
    log.info("Replayed key={} to topic={}", record.key(), originalTopic);
}

Replay is not automatically idempotent. Replaying a DLT message re-triggers your listener’s full business logic — if that logic already partially succeeded before failing (e.g. it charged a payment, then failed sending a confirmation event), a naive replay can duplicate the side effect. Design listeners to be idempotent (dedupe by message key/ID) if DLT replay is part of your recovery process.

6. Non-blocking retry with @RetryableTopic

DefaultErrorHandler‘s backoff (§2) is blocking — the consumer thread stalls on that partition for the entire backoff duration, delaying every other message behind it in the same partition. For longer backoff windows or high-throughput topics, Spring Kafka’s @RetryableTopic annotation implements non-blocking retry: failed messages are republished to dedicated retry topics, freeing the original partition to keep processing while the retry delay elapses elsewhere.

@RetryableTopic(
    attempts = "4",
    backoff = @Backoff(delay = 1000, multiplier = 2.0, maxDelay = 10000),
    autoCreateTopics = "true",
    include = {TransientDataAccessException.class, ResourceAccessException.class}
)
@KafkaListener(topics = "orders", groupId = "order-service")
public void consume(ConsumerRecord<String, String> record) {
    processOrder(record.value());
}

@DltHandler
public void handleDlt(ConsumerRecord<String, String> record) {
    log.error("Exhausted retries for key={}", record.key());
    alertingService.notify(record);
}

What this creates automatically:

orders → orders-retry-0 (1s delay) → orders-retry-1 (2s delay) → orders-retry-2 (4s delay) → orders.DLT

Each retry attempt happens on its own topic and partition, consumed by a dedicated internal listener that respects the configured delay before reprocessing — the original orders topic’s consumer thread is never blocked waiting.

7. Blocking vs non-blocking retry — which to use

Decision rule: if backoff intervals are short (seconds) and throughput is moderate, blocking retry’s simplicity usually wins. Once total backoff time starts meaningfully delaying unrelated messages behind a struggling one — or a downstream outage means minutes-long backoffs — non-blocking retry topics prevent one bad message from stalling an entire partition’s throughput.

8. Monitoring error handling health

# Treat the DLT like any other topic for lag monitoring
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group orders-dlt-monitor

Alerting practice: alert on DLT message rate (messages/minute), not just absolute count — a slow trickle over days can be normal for a large system, but a sudden rate spike means something just broke and needs immediate attention.

9. Testing error handler behavior

@SpringBootTest
@EmbeddedKafka(partitions = 1, topics = {"orders", "orders.DLT"})
class ErrorHandlerIntegrationTest {

    @Autowired KafkaTemplate<String, String> kafkaTemplate;
    @Autowired ConsumerFactory<String, String> consumerFactory;

    @Test
    void nonRetryableExceptionGoesStraightToDlt() {
        kafkaTemplate.send("orders", "bad-key", "malformed-payload");

        var dltConsumer = consumerFactory.createConsumer("dlt-verify", "test");
        dltConsumer.subscribe(List.of("orders.DLT"));

        var records = KafkaTestUtils.getRecords(dltConsumer, Duration.ofSeconds(10));
        assertThat(records.count()).isEqualTo(1);
        assertThat(records.iterator().next().headers().lastHeader("kafka_dlt-exception-fqcn"))
            .isNotNull();
    }

    @Test
    void retryableExceptionSucceedsWithinBackoffWindow() {
        // simulate a transient failure that succeeds on the 2nd attempt
        // assert the message is processed successfully and NEVER reaches the DLT
    }
}

Test both directions explicitly: a non-retryable exception should land on the DLT immediately (verify the exception headers are set correctly for downstream alerting), and a transient failure that later succeeds should never reach the DLT at all — both are easy to get backwards with a misconfigured addNotRetryableExceptions list.

10. Common pitfalls

Using blocking retry with long backoff windows on a busy topic — stalls the entire partition behind one struggling message; reach for @RetryableTopic once backoff durations get long relative to your throughput needs
Forgetting include/addNotRetryableExceptions and retrying everything — a permanent bug (bad deserialization, null pointer) gets retried N times for no benefit before finally landing on the DLT, adding latency with zero chance of success
Assuming DLT replay is safe by default — as noted in §5, replay can duplicate side effects if listener logic isn’t idempotent
Not monitoring the DLT at all — the single most common production gap; a DLT with no consumer and no alerting is a silent failure pipeline that nobody notices until a customer complains
Ordering assumptions with @RetryableTopic — non-blocking retry topics don’t preserve strict per-key ordering the way blocking retry does; don’t use it for use cases where retry-time ordering matters as much as eventual delivery

Key Takeaways

DefaultErrorHandler: retry N times with backoff, then publish to DLT — but this blocks the partition for the backoff duration
DeadLetterPublishingRecoverer routes failures to {topic}.DLT with full error headers
Exponential backoff: 1s → 2s → 4s — avoids hammering a struggling downstream service
Non-retryable exceptions (deserialization, validation) skip retries, go straight to DLT
@RetryableTopic gives non-blocking retry via dedicated retry topics — trades strict ordering for not stalling the partition during backoff
Always monitor the DLT — track message rate, not just count, and alert on spikes
DLT messages can be replayed after a bug fix, but replay isn’t automatically idempotent — design for it explicitly
Test both directions: non-retryable exceptions must reach the DLT quickly; transient ones that later succeed must never reach it

Support me through GitHub Sponsors.

➡️ Day 19: Kafka vs RabbitMQ — when to replace your queue

Resources

📘 Kafka: The Definitive Guide — Chapter 5 (Reliable Data Delivery)
🌐 docs.spring.io/spring-kafka/reference — Error handling
🌐 Spring Kafka — Non-Blocking Retries — @RetryableTopic reference

👉 Link to Medium blog