Kafka

60-Day Kafka 4 Learning Plan · Week 8 — Day 53 of 60

Editorial Team

60-Day Kafka 4 Learning Plan · Week 8 — Production & Cloud Sources: Kafka: The Definitive Guide Ch.2 · confluent.io/confluent-cloud · docs.aws.amazon.com/msk · aiven.io/kafka

Goal

Understand the trade-offs between self-hosted and managed Kafka, compare the three major managed providers with their current (not stale) Kafka 4 support status, connect Spring Boot (labs-api) to each one with minimal configuration changes, and weigh total cost of ownership and vendor lock-in beyond the feature checklist.

1. Self-hosted vs managed — the trade-off

Rule of thumb: go managed until operational complexity becomes the bottleneck. Migrate self-hosted when you need cost control or deep customisation at scale.

2. Provider comparison

The Kafka-version row above is stale — see §7 for the current picture as of this course’s writing (August 2026). Both Confluent Cloud/Platform and Amazon MSK have supported Kafka 4.0 (and, for MSK, 4.1) for well over a year by this point. Managed-service version support is exactly the kind of fast-moving fact that’s easy to get wrong by copying an older comparison table — always verify current support against the provider’s own release notes before making a platform decision based on version availability.

3. Confluent Cloud — Spring Boot config

Get credentials

  1. Confluent Cloud → Environments → Cluster → API Keys → Create key
  2. Copy API Key (username) and API Secret (password)
  3. Copy bootstrap server URL from Cluster → Cluster Settings
# application.yml — labs-api on Confluent Cloud
spring:
  kafka:
    bootstrap-servers: pkc-xxxxx.us-east-1.aws.confluent.cloud:9092
    security.protocol: SASL_SSL
    sasl.mechanism: PLAIN
    sasl.jaas.config: >-
      org.apache.kafka.common.security.plain.PlainLoginModule required
      username="${CONFLUENT_API_KEY}"
      password="${CONFLUENT_API_SECRET}";

    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer
      acks: all

    consumer:
      group-id: labs-api-group
      auto-offset-reset: earliest

Environment variables

CONFLUENT_API_KEY=ABCDEFGHIJKLMNOP
CONFLUENT_API_SECRET=abcdefghijklmnopqrstuvwxyz1234567890abcdefghij

Confluent CLI quickstart

# Install Confluent CLI
curl -sL --http1.1 https://cnfl.io/cli | sh -s -- latest

# Login and create a topic
confluent login
confluent kafka topic create labs.events --partitions 6

# Produce test message
confluent kafka topic produce labs.events

4. Amazon MSK — Spring Boot config (IAM auth)

MSK supports three auth methods: unauthenticated (dev only), SASL/SCRAM, and IAM. IAM is the recommended production method for AWS-native stacks — no passwords to manage.

Maven dependency for IAM auth

<dependency>
  <groupId>software.amazon.msk</groupId>
  <artifactId>aws-msk-iam-auth</artifactId>
  <version>2.1.1</version>
</dependency>

application.yml — IAM authentication

spring:
  kafka:
    bootstrap-servers: b-1.labs.xxxxx.c3.kafka.us-east-1.amazonaws.com:9098
    security.protocol: SASL_SSL
    sasl.mechanism: AWS_MSK_IAM
    sasl.jaas.config: >-
      software.amazon.msk.auth.iam.IAMLoginModule required
      awsRoleArn="arn:aws:iam::123456789012:role/labs-kafka-role";
    properties:
      sasl.client.callback.handler.class: >-
        software.amazon.msk.auth.iam.IAMClientCallbackHandler

IAM policy for the ECS/EC2 role

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kafka-cluster:Connect",
        "kafka-cluster:DescribeCluster"
      ],
      "Resource": "arn:aws:kafka:us-east-1:123456789012:cluster/labs/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "kafka-cluster:ReadData",
        "kafka-cluster:WriteData",
        "kafka-cluster:DescribeTopic"
      ],
      "Resource": "arn:aws:kafka:us-east-1:123456789012:topic/labs/*/labs.events"
    }
  ]
}

application.yml — SASL/SCRAM (alternative to IAM)

spring:
  kafka:
    bootstrap-servers: b-1.labs.xxxxx.c3.kafka.us-east-1.amazonaws.com:9096
    security.protocol: SASL_SSL
    sasl.mechanism: SCRAM-SHA-512
    sasl.jaas.config: >-
      org.apache.kafka.common.security.scram.ScramLoginModule required
      username="${MSK_USERNAME}"
      password="${MSK_PASSWORD}";

5. Aiven — Spring Boot config

# application.yml — labs-api on Aiven for Apache Kafka
spring:
  kafka:
    bootstrap-servers: labs-kafka.aivencloud.com:26752
    security.protocol: SSL
    ssl:
      key-store-type: PKCS12
      key-store: classpath:client.keystore.p12     # download from Aiven console
      key-store-password: ${AIVEN_KEYSTORE_PASSWORD}
      trust-store-type: JKS
      trust-store: classpath:client.truststore.jks
      trust-store-password: ${AIVEN_TRUSTSTORE_PASSWORD}

Aiven uses mTLS (mutual TLS) by default — download the keystore from the Aiven console under your service’s Connection Information tab.

6. Decision guide

7. Correcting the version table — actual Kafka 4 support status

§2’s version row (“Confluent Cloud: Latest (3.x)”, “Amazon MSK: 2.8–3.x (lags)”) doesn’t reflect where these providers actually stand. Verified current status:

  • Amazon MSK added Kafka 4.0 support broadly across AWS regions, bringing KIP-848’s new consumer rebalance protocol and the Java 17 requirement that comes with it, and later added Kafka 4.1 support, introducing early-access Queues (KIP-932) and a Streams Rebalance Protocol extensionbuilding on the 4.0 consumer rebalance work. The “MSK lags behind” reputation from earlier Kafka major-version eras no longer holds by the time of Kafka 4.
  • Confluent shipped Confluent Platform 8.0 built directly on Apache Kafka 4.0, and Confluent Cloud’s underlying Kora engine maintains full protocol compatibility with Kafka, including the same KIP-896 client-API-version baseline changes that came with Kafka 4.0.
  • Aiven for Apache Kafka generally tracks recent open-source Kafka releases closely as a core part of its value proposition — verify the exact current version against Aiven’s own service catalog before committing, since exact version cadence can shift.

The practical lesson, beyond just fixing this one table: a static comparison table copied into a course or internal wiki is a snapshot that goes stale — especially for managed-service version support, which changes on each provider’s own release cadence, not the course author’s. Before making a platform decision that hinges on version availability, always check the provider’s current release notes/supported-versions page directly rather than trusting any table (including this corrected one) as permanently accurate.

8. Total cost of ownership — beyond the feature checklist

§1’s “Cost at scale: Lower (self-hosted) / Higher (managed)” is directionally true but too coarse to actually budget from — each managed provider’s cost model has its own specific levers worth understanding before committing.

Why this deserves its own line item in a decision process, not just “managed costs more”: the Day 51 sizing exercise (partition count, replication factor, retention, throughput) translates directly into each provider’s specific cost model — the same workload can cost meaningfully different amounts across providers depending on which cost dimension it stresses most (a partition-count-heavy workload vs a storage-heavy one vs a cross-AZ-replication-heavy one). Run the actual numbers through each provider’s pricing calculator using Day 51’s sizing output, not a generic “managed is more expensive” assumption.

9. Vendor lock-in and migration considerations

Beyond the feature and cost comparisons, worth weighing explicitly: how hard would it be to leave, if that ever became necessary?

  • Protocol-level portability is generally good — all three providers speak standard Kafka wire protocol, so a Spring Boot client (§3-§5) mostly just changes bootstrap-servers and auth config to switch providers, as this material’s own consistent Spring config pattern across sections demonstrates.
  • Provider-specific features are the real lock-in risk — Confluent Cloud’s ksqlDB and Stream Governance, MSK’s IAM-based auth model and tight AWS service integration (Day 45’s ACL model doesn’t map 1:1 onto IAM policies), and any provider-specific connector catalog entries all represent genuine migration friction if used.
  • MirrorMaker 2 (Day 54, next in this course) is the standard tool for actually moving data between clusters/providers when a migration is needed — worth knowing it exists as an option before assuming a provider switch means a from-scratch data migration.

Practical framing: treat “how portable is this if we need to switch” as a real evaluation criterion alongside features and cost, especially for a team early in its Kafka journey that might reasonably outgrow its first provider choice. Leaning heavily on provider-specific conveniences (ksqlDB, IAM-integrated ACLs) is a legitimate choice — just make it a conscious one, not a default that quietly increases switching cost over time.

10 Common pitfalls

  • Trusting a static Kafka-version comparison table without checking current provider release notes — managed-service version support is genuinely fast-moving and easy to get stale (§7)
  • Budgeting “managed costs more” without running the actual sizing numbers through each provider’s specific cost model — CKUs, broker-hours + cross-AZ transfer, and per-GB storage all respond differently to the same workload characteristics (§8)
  • Adopting provider-specific conveniences (ksqlDB, IAM ACLs) without recognizing the migration cost they create — not wrong to use them, but worth being a deliberate choice rather than an unconsidered default (§9)
  • Assuming MSK “lags” on Kafka versions based on older reputation — current MSK has kept pace with Kafka 4.0/4.1 within months of upstream release (§7)
  • Skipping the Day 51 sizing exercise when evaluating managed providers — capacity planning inputs (partitions, RF, retention, throughput) are exactly what drive real managed-service cost, not just a feature checklist

Key Takeaways

  • Managed Kafka = no broker ops — provider handles upgrades, scaling, HA, and monitoring
  • Confluent Cloud: richest ecosystem (Schema Registry, ksqlDB, Connect) — best Kafka experience, and Kora tracks Kafka 4.0 protocol compatibility
  • Amazon MSK: native AWS IAM auth, tight VPC integration — best for AWS-native teams, and has supported Kafka 4.0/4.1 for well over a year as of this writing, not “2.8–3.x lagging”
  • Aiven: true multi-cloud, EU data residency, open-source — best for cloud-agnostic stacks
  • Spring Boot config is largely portable across providers — mostly bootstrap-servers + auth config changes, though provider-specific features (ksqlDB, IAM ACLs) create real migration friction if adopted
  • Run Day 51’s sizing exercise through each provider’s specific cost model (CKUs, broker-hours + cross-AZ transfer, per-GB storage) before comparing costs — the same workload can cost meaningfully different amounts depending on which cost dimension it stresses
  • Always verify current managed-service version support against the provider’s own release notes — comparison tables (including this one) go stale

Support me through GitHub Sponsors.

Thank you for Reading !! See you in the next post.

Next

➡️ Day 54: MirrorMaker 2 — cross-cluster replication setup

Resources

Link to Medium blog

, ,

Related Posts

Kafka

60-Day Kafka 4 Learning Plan · Week 9 — Day 60 of 60

Editorial Team
Kafka

60-Day Kafka 4 Learning Plan · Week 9 — Day 59 of 60

Editorial Team