60-Day Kafka 4 Learning Plan · Week 3 — Day 19 of 60

60-Day Kafka 4 Learning Plan · Week 3 — Spring Boot Integration Sources: Kafka: The Definitive Guide Ch.1 · kafka.apache.org/43/documentation

Goal

Understand the fundamental architectural difference between Kafka and RabbitMQ, know when to choose each, and learn how to migrate incrementally — including the pitfalls of dual-publish and how to verify the migration is actually safe.

1. The fundamental difference: log vs queue

RabbitMQ — Message Queue

Messages are deleted after consumption — no retention
Push-based: broker pushes messages to subscribed consumers
Flexible routing: exchanges (topic, fanout, direct, headers), bindings, routing keys
Competing consumers: one consumer per message within a queue
Great for: task queues, RPC patterns, background job processing, workflows

Kafka — Distributed Log ★

Messages are retained for a configurable period (days, weeks, forever)
Pull-based: consumers control their own pace by polling
Ordered append-only log per partition
Multiple independent consumer groups: every group gets every message
Great for: event streaming, audit trails, replay, analytics, stream processing

The single most important question: do you need to replay messages? If yes → Kafka. If message deletion on consumption is fine → RabbitMQ may be simpler.

2. Head-to-head comparison

3. Choose Kafka when…

✅ Event replay — audits, re-processing after bug fixes, onboarding new consumers to historical data
✅ Fan-out — multiple independent services must consume the same events
✅ High throughput — exceeds ~100k msg/s or significant growth expected
✅ Event sourcing — durable event log as the source of truth for state reconstruction
✅ Stream processing — aggregations, joins, windowing (Kafka Streams, Apache Flink)
✅ Long retention — days/weeks of history for analytics, ML pipelines, or compliance
✅ Event-driven microservices — decoupled producers and consumers with no direct dependency

4. Keep RabbitMQ when…

✅ Simple task queue — one producer, one worker pool, process and discard
✅ Complex routing — dynamic topic/fanout/header exchanges with runtime bindings
✅ RPC patterns — request/reply with correlation IDs and temporary reply queues
✅ Lower volume — < 100k msg/s with simple delivery guarantees
✅ Existing investment — team is proficient in AMQP, no business case for replatforming
⚠️ Per-queue ordering + per-consumer acknowledgement matters more than replay

5. Migration pattern: RabbitMQ → Kafka

Migrate incrementally — never big-bang switch.

Step 1 — Dual-publish

Write to both RabbitMQ and Kafka simultaneously. Validate message parity between both systems. Ensures zero data loss during transition.

// In your producer service
rabbitTemplate.convertAndSend(exchange, routingKey, event);   // existing
kafkaTemplate.send("orders", event.getId(), event);           // new

Step 2 — Migrate consumers one by one

Move services from RabbitMQ consumers to @KafkaListener one at a time. Allows per-service rollback if problems arise.

// Old (RabbitMQ)
@RabbitListener(queues = "orders.queue")
public void consumeFromRabbit(OrderEvent event) { ... }

// New (Kafka)
@KafkaListener(topics = "orders", groupId = "order-service")
public void consumeFromKafka(ConsumerRecord<String, OrderEvent> record) { ... }

Step 3 — Stop publishing to RabbitMQ

Once all consumers have migrated, stop the dual-publish. Drain the remaining Rabbit queue, then decommission.

Step 4 — Leverage Kafka replay

Onboard new services by replaying from the beginning of the Kafka log — no need to re-run original producers. This is impossible with RabbitMQ.

// Replay from the start for a new service
@KafkaListener(topics = "orders", groupId = "new-analytics-service")
// Spring auto-resets to earliest if no committed offset found
// spring.kafka.consumer.auto-offset-reset: earliest

6. Dual-publish pitfalls — and the transactional outbox alternative

The naive dual-publish in §5 Step 1 has a real correctness gap: the two send() calls are not atomic. If the app crashes (or either broker rejects the write) between the two calls, RabbitMQ and Kafka silently diverge — exactly the “message parity” problem the migration step was supposed to prevent.

rabbitTemplate.convertAndSend(...);   // succeeds
                                       // ← app crashes HERE
kafkaTemplate.send(...);              // never happens — Kafka is now missing this event

Transactional outbox pattern — the production-grade fix: write the event to an outbox table in the same database transaction as the business change, then a separate relay process reads the outbox and publishes to both brokers, retrying independently per broker until each confirms.

@Transactional
public void createOrder(Order order) {
    orderRepository.save(order);
    outboxRepository.save(new OutboxEvent("orders", order.getId(), toJson(order)));
    // Both writes commit atomically as one DB transaction — no partial-publish gap
}

@Scheduled(fixedDelay = 1000)
public void relayOutbox() {
    List<OutboxEvent> pending = outboxRepository.findUnpublished();
    for (OutboxEvent event : pending) {
        rabbitTemplate.convertAndSend(event.exchange(), event.routingKey(), event.payload());
        kafkaTemplate.send(event.topic(), event.key(), event.payload());
        outboxRepository.markPublished(event.id()); // only after BOTH succeed
    }
}

Trade-off: the outbox adds a database table, a relay process, and slightly more publish latency (bounded by the relay’s polling interval) — but it’s the difference between “message parity during migration” being an actual guarantee versus a best-effort hope. For a short migration window with low risk tolerance for silent gaps, it’s worth the extra plumbing.

7. Consumer acknowledgement models compared

The two systems’ ack models aren’t just different APIs — they reflect the queue-vs-log distinction directly, and this matters when migrating consumer logic, not just producer logic.

Migration gotcha: RabbitMQ code that selectively acks/nacks individual messages out of delivery order has no direct Kafka equivalent — Kafka’s offset model is fundamentally sequential. Consumer logic relying on per-message selective ack needs to be redesigned around either strict per-partition processing order or an explicit skip/DLT strategy (Day 18) rather than ported line-for-line.

8. RabbitMQ Streams — the convergence feature worth knowing

Modern RabbitMQ (3.9+) ships a Streams plugin that adds a Kafka-like append-only, replayable log structure on top of RabbitMQ — non-destructive reads, configurable retention, and consumer offset tracking similar to Kafka’s model.

Why this matters for the decision in §3/§4: if the only reason pulling you toward Kafka is “we need replay” but the rest of your architecture (routing, RPC patterns, existing AMQP investment) fits RabbitMQ well, RabbitMQ Streams can close that specific gap without a full platform migration. It’s not a full Kafka Streams/ksqlDB-equivalent processing engine, but for straightforward replay/retention needs it’s worth evaluating before committing to a Kafka migration.

Still choose Kafka over RabbitMQ Streams when: you need genuine high-throughput partitioned parallelism across many consumer groups, native stream processing (joins, windowing, aggregations via Kafka Streams), or you’re already standardizing on Kafka elsewhere in the org — operational consistency across services often outweighs a feature-by-feature comparison for a single use case.

9. Monitoring migration health — verifying message parity

Dual-publish (§5, ideally via the outbox in §6) is only safe if you actively verify parity — don’t assume it’s working silently.

@Component
@Slf4j
public class ParityMonitor {

    @Scheduled(fixedDelay = 60000)
    public void checkParity() {
        long rabbitCount = rabbitAdmin.getQueueInfo("orders.queue").getMessageCount();
        long kafkaTopicEndOffset = kafkaAdminClient
            .listOffsets(Map.of(new TopicPartition("orders", 0), OffsetSpec.latest()))
            .partitionResult(new TopicPartition("orders", 0)).get().offset();

        // Compare against expected counts/checksums per time window — exact equality
        // rarely holds due to timing, but a growing divergence trend is the real signal
        if (divergenceExceedsThreshold(rabbitCount, kafkaTopicEndOffset)) {
            alertingService.notify("RabbitMQ/Kafka parity divergence detected");
        }
    }
}

What to actually alert on: raw message counts drift naturally due to timing (in-flight messages, different consumption speeds) — track a trend over rolling windows rather than expecting instant equality, and alert when divergence grows rather than on any single non-zero delta.

10. Testing during migration

@SpringBootTest
@EmbeddedKafka(partitions = 1, topics = "orders")
class DualPublishParityTest {

    @Autowired OutboxRelay outboxRelay;
    @MockBean RabbitTemplate rabbitTemplate; // verify without a real broker
    @Autowired KafkaTemplate<String, String> kafkaTemplate;

    @Test
    void publishesToBothBrokersForSameEvent() {
        outboxRelay.relayOutbox(); // drains a seeded outbox event

        verify(rabbitTemplate).convertAndSend(eq("orders.exchange"), anyString(), any());
        // plus a Kafka-side assertion via @EmbeddedKafka consumer, confirming the SAME event id
        // landed on both — this is what actually proves parity, not just "both were called"
    }

    @Test
    void relayRetriesIndependentlyPerBrokerOnFailure() {
        doThrow(new AmqpException("down")).when(rabbitTemplate).convertAndSend(any(), any(), any());
        outboxRelay.relayOutbox();
        // assert Kafka still received the event even though RabbitMQ failed —
        // and assert the outbox entry is NOT marked published until both succeed
    }
}

The valuable test here isn’t “was send() called” — it’s confirming the same logical event ID appears on both systems, and that a partial failure doesn’t mark the outbox entry complete. That’s the actual correctness property the migration depends on.

11. Common pitfalls

Naive dual-publish without an outbox — silent divergence on any crash between the two send() calls, discovered only much later (often by a customer, not a dashboard)
Porting selective-ack RabbitMQ consumer logic line-for-line to Kafka — the semantics don’t map; needs redesign around sequential offset commits and DLT handling instead
Trusting raw message-count parity checks — natural timing drift makes exact equality checks noisy; alert on trend, not instantaneous mismatch
Migrating all consumers at once “to save time” — defeats the entire point of the incremental Step 2 approach; per-service rollback becomes impossible if everything moves together
Choosing Kafka purely for replay without checking RabbitMQ Streams first — a full platform migration is expensive; if replay is genuinely the only gap, evaluate the lighter-weight option before committing

12. Spring Boot: run both side by side

<!-- pom.xml -->
<dependency>
  <groupId>org.springframework.kafka</groupId>
  <artifactId>spring-kafka</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-amqp</artifactId>
</dependency>

They coexist cleanly — use each where it excels in the same application during migration or permanently.

Key Takeaways

Kafka is a log; RabbitMQ is a queue — fundamentally different retention models
Choose Kafka for replay, fan-out, high throughput, event sourcing, stream processing
Keep RabbitMQ for task queues, RPC patterns, complex routing, lower volume — and consider RabbitMQ Streams if replay is the only gap
They can coexist — use each where it excels in the same architecture
Migrate incrementally with dual-publish — never big-bang switch — and back it with a transactional outbox, not naive back-to-back send() calls
RabbitMQ’s per-message selective ack has no direct Kafka equivalent — consumer logic needs redesigning, not porting
Verify parity actively during migration; track divergence trend, don’t assume dual-publish is silently correct
Spring Boot supports both: spring-kafka and spring-amqp side-by-side

Support me through GitHub Sponsors.

➡️ Day 20 — Multi-service: labs-api ↔ labs-socket via Kafka

Resources

📘 Kafka: The Definitive Guide — Chapter 1 (Meet Kafka)
🌐 kafka.apache.org/43/documentation
🌐 rabbitmq.com/docs
🌐 RabbitMQ Streams overview

👉 Link to Medium blog