60-Day Kafka 4 Learning Plan · Week 3 — Day 15 of 60

60-Day Kafka 4 Learning Plan · Week 3 — Spring Boot Integration Sources: Kafka: The Definitive Guide Ch.3 & 4 · docs.spring.io/spring-kafka/reference

Goal

Understand how the spring-kafka starter auto-configures producer and consumer infrastructure, write your first KafkaTemplate and @KafkaListener, manage topics declaratively via KafkaAdmin, structure multi-environment config with Spring profiles, and wire in health checks, testing, and observability.

1. Why Spring Kafka instead of raw clients?

Auto-configures producer, consumer, and listener infrastructure from application.yml — no manual KafkaProducer/KafkaConsumer wiring. Annotations + properties do the work.

Spring Kafka handles thread pools, error handling, retry, offset commit strategy, and graceful shutdown for you. It integrates with Spring’s dependency injection, testing support (@EmbeddedKafka), and Actuator health checks.

2. pom.xml — single starter dependency

<dependency>
  <groupId>org.springframework.kafka</groupId>
  <artifactId>spring-kafka</artifactId>
</dependency>

That’s it — version is managed by Spring Boot’s dependency management (BOM), matching the Spring Boot 3.x release train.

3. Auto-configured beans from application.yml

application.yml (spring.kafka.* properties)
        │
        ▼
ProducerFactory → KafkaTemplate          ConsumerFactory → ListenerContainerFactory
        │                                          │
        ▼                                          ▼
@Autowired KafkaTemplate.send()          @KafkaListener annotated method

Spring Boot’s KafkaAutoConfiguration reads spring.kafka.* properties and registers:

ProducerFactory and KafkaTemplate<K, V>
ConsumerFactory and ConcurrentKafkaListenerContainerFactory
KafkaAdmin for topic management
KafkaTransactionManager (if transactions are configured)

4. Minimal application.yml

spring:
  kafka:
    bootstrap-servers: localhost:9092
    consumer:
      group-id: order-service
      auto-offset-reset: earliest
    producer:
      # key/value serializers default to StringSerializer

That’s enough to send and receive string messages. Override serializers, batch settings, security, etc. as needed (covered Days 16–18).

5. KafkaTemplate — inject and send

@Service
@RequiredArgsConstructor
public class OrderProducer {

    private final KafkaTemplate<String, String> kafkaTemplate;

    public void send(String orderId, String payload) {
        kafkaTemplate.send("orders", orderId, payload);
    }
}

KafkaTemplate.send() is asynchronous and returns a CompletableFuture<SendResult<K, V>>:

public void sendWithCallback(String orderId, String payload) {
    kafkaTemplate.send("orders", orderId, payload)
        .whenComplete((result, ex) -> {
            if (ex == null) {
                log.info("Sent to partition={} offset={}",
                    result.getRecordMetadata().partition(),
                    result.getRecordMetadata().offset());
            } else {
                log.error("Send failed", ex);
            }
        });
}

6. @KafkaListener — receive without manual polling

@Component
@Slf4j
public class OrderConsumer {

    @KafkaListener(topics = "orders", groupId = "order-service")
    public void consume(ConsumerRecord<String, String> record) {
        log.info("key={} value={} partition={}",
            record.key(), record.value(), record.partition());
    }
}

Spring auto-wires polling, deserialization, offset commit, error handling, and graceful shutdown. You only write the business logic inside the method body.

Other useful listener parameter types

@KafkaListener(topics = "orders")
public void consume(
        @Payload String value,
        @Header(KafkaHeaders.RECEIVED_KEY) String key,
        @Header(KafkaHeaders.RECEIVED_PARTITION) int partition,
        @Header(KafkaHeaders.OFFSET) long offset,
        Acknowledgment ack) {

    log.info("key={} value={} partition={} offset={}", key, value, partition, offset);
    ack.acknowledge();  // manual commit — requires ack-mode: MANUAL
}

7. KafkaAdmin — declarative topic management

KafkaAdmin (auto-configured from spring.kafka.bootstrap-servers) lets you define topics as Spring beans instead of relying on broker auto-creation or manual CLI commands. On startup, Spring reconciles any NewTopic beans against the cluster — creating missing topics, and (with KafkaAdmin.setModifyTopicConfigs(true)) reconciling config drift on existing ones.

@Configuration
public class KafkaTopicConfig {

    @Bean
    public NewTopic ordersTopic() {
        return TopicBuilder.name("orders")
                .partitions(6)
                .replicas(3)
                .config(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, "2")
                .build();
    }
}

spring:
  kafka:
    admin:
      fail-fast: true   # fail app startup if topic creation fails, instead of silently continuing

Production practice: disable broker-side auto.create.topics.enable and rely on KafkaAdmin beans (or a separate infra pipeline) instead. Broker auto-creation silently accepts typos as new topics with default (often wrong) partition/replication settings — a classic source of “why does this topic have 1 partition and RF=1 in production” incidents.

8. Health checks — Spring Boot Actuator integration

With spring-boot-starter-actuator on the classpath, Spring Kafka automatically contributes a kafka health indicator that checks cluster connectivity via KafkaAdmin.describeCluster().

management:
  endpoint:
    health:
      show-details: always
  health:
    kafka:
      enabled: true

curl localhost:8080/actuator/health
# { "status": "UP", "components": { "kafka": { "status": "UP" } } }

Caveat: the default Kafka health indicator only confirms the cluster is reachable — it does not verify your specific topics exist, that your consumer groups are actively consuming, or that lag is healthy. For real production readiness, pair it with the lag monitoring covered in Day 6, not as a replacement for it.

9. Testing with @EmbeddedKafka

@SpringBootTest
@EmbeddedKafka(partitions = 3, topics = "orders")
class OrderProducerIntegrationTest {

    @Autowired KafkaTemplate<String, String> kafkaTemplate;

    @Autowired
    EmbeddedKafkaBroker embeddedKafkaBroker;

    @Test
    void sendsOrderSuccessfully() throws Exception {
        var consumer = new DefaultKafkaConsumerFactory<String, String>(
            KafkaTestUtils.consumerProps("test-group", "true", embeddedKafkaBroker))
            .createConsumer();
        embeddedKafkaBroker.consumeFromAnEmbeddedTopic(consumer, "orders");

        kafkaTemplate.send("orders", "o1", "payload").get(5, TimeUnit.SECONDS);

        ConsumerRecord<String, String> received =
            KafkaTestUtils.getSingleRecord(consumer, "orders", Duration.ofSeconds(5));
        assertThat(received.value()).isEqualTo("payload");
    }
}

@EmbeddedKafka spins up an in-memory broker for the test class — no Docker or external cluster needed. It’s the standard way to integration-test producers, listeners, and topic configuration together, faster and more reliably than pointing tests at a shared dev cluster.

10. Multiple KafkaTemplate / listener factories for different types

A real application often needs more than one message shape (e.g. String events plus typed Order/Invoice JSON payloads). Spring Boot’s auto-configuration wires exactly one default KafkaTemplate and one default listener container factory — additional ones must be defined explicitly and referenced by name.

@Bean
public KafkaTemplate<String, Order> orderKafkaTemplate(ProducerFactory<String, Order> pf) {
    return new KafkaTemplate<>(pf);
}

@Bean
public ConcurrentKafkaListenerContainerFactory<String, Order> orderListenerFactory(
        ConsumerFactory<String, Order> cf) {
    var factory = new ConcurrentKafkaListenerContainerFactory<String, Order>();
    factory.setConsumerFactory(cf);
    return factory;
}

@KafkaListener(topics = "orders", containerFactory = "orderListenerFactory")
public void consume(Order order) { ... }

Naming matters: when multiple KafkaTemplate beans exist, @Autowired KafkaTemplate<String, String> becomes ambiguous — use @Qualifier or distinct bean names, and always specify containerFactory explicitly on @KafkaListener once you have more than one factory. Relying on “whichever bean Spring happens to autowire” is a common source of subtle misconfiguration.

11. Observability — Micrometer metrics & tracing

Spring Kafka auto-registers Micrometer metrics for producers, consumers, and listener containers when micrometer-core is on the classpath — no manual KafkaClientMetrics binding needed for the common case (unlike the manual binding pattern from earlier lessons, which is still useful for custom/non-Spring-managed clients).

management:
  metrics:
    enable:
      kafka: true
spring:
  kafka:
    listener:
      observation-enabled: true   # emits Micrometer Observation spans per listener invocation
    producer:
      properties:
        # combined with observation-enabled, integrates with distributed tracing (Micrometer Tracing / OTel)

With observation-enabled: true and a tracing bridge (e.g. Micrometer Tracing + OpenTelemetry) on the classpath, each @KafkaListener invocation and KafkaTemplate.send() becomes a traced span — letting you follow a message end-to-end alongside HTTP/DB spans in the same trace, which is invaluable for debugging cross-service event flows.

12. Common pitfalls

Relying on broker auto.create.topics.enable instead of KafkaAdmin beans — a producer typo becomes a permanent misconfigured topic in production
Forgetting containerFactory when multiple listener factories exist — Spring silently uses the default factory’s settings (serializers, concurrency, ack-mode), which may be wrong for that listener’s actual payload type
Testing against a shared dev Kafka cluster instead of @EmbeddedKafka — flaky tests from shared state, topic pollution across test runs, and slower CI feedback loops
Treating the Actuator kafka health check as sufficient production readiness — it only proves broker connectivity, not that your consumers are healthy or your lag is under control
Not setting admin.fail-fast: true — a broken NewTopic config (bad partition count, invalid replica assignment) can fail silently on startup, leaving the app running against a topic that doesn’t match what the code expects

13. Spring profiles for multi-environment Kafka config

application-dev.yml

spring:
  kafka:
    bootstrap-servers: localhost:9092
    # No security, single broker

application-prod.yml

spring:
  kafka:
    bootstrap-servers: kafka-1:9093,kafka-2:9093,kafka-3:9093
    properties:
      security.protocol: SASL_SSL
      sasl.mechanism: SCRAM-SHA-512
    producer:
      acks: all

Activate a profile

java -jar app.jar --spring.profiles.active=prod

Or via environment variable (common in containers):

export SPRING_PROFILES_ACTIVE=prod

Spring merges application.yml (shared defaults) with application-{profile}.yml (overrides), so you only need to specify what differs per environment.

Key Takeaways

spring-kafka starter auto-configures ProducerFactory, ConsumerFactory, KafkaTemplate
All Kafka config lives under spring.kafka.* in application.yml — no boilerplate Properties objects
KafkaTemplate.send() handles serialization and async delivery with a returned CompletableFuture
@KafkaListener replaces manual poll() loops — Spring manages the consumer lifecycle
Prefer KafkaAdmin + NewTopic beans over broker auto-creation — declarative, reviewable, fails fast on misconfiguration
The Actuator kafka health indicator checks connectivity only — pair it with real lag monitoring for production readiness
@EmbeddedKafka is the standard way to integration-test Kafka code without an external cluster
Multiple payload types need explicit, named KafkaTemplate/listener factory beans — don’t rely on the single auto-configured default
observation-enabled: true + a tracing bridge gets you distributed tracing across Kafka hops for free
Spring profiles (dev/staging/prod) cleanly separate environment-specific Kafka config
Spring Boot 3.x requires Java 17+ and works natively with Kafka 4 KRaft brokers

Support me through GitHub Sponsors.

➡️ Day 16: Producer config — acks, retries, idempotence

Resources

📘 Kafka: The Definitive Guide — Chapter 3 & 4 (Producers & Consumers)
🌐 docs.spring.io/spring-kafka/reference — Spring for Apache Kafka
🌐 Spring Kafka — Testing — @EmbeddedKafka reference

👉 Link to Medium blog