60-Day Kafka 4 Learning Plan · Week 7 — Day 43 of 60

60-Day Kafka 4 Learning Plan · Week 7 — Security & Monitoring Sources: Kafka: The Definitive Guide Ch.12 · kafka.apache.org/documentation/#security_ssl

Goal

Secure Kafka wire traffic with TLS. Understand CA, keystore, and truststore roles; generate a self-signed CA and broker certificates with keytool; configure the Kafka broker with an SSL listener (including inter-broker traffic); wire Spring Boot (labs-api) to connect over SSL; and understand hostname verification, certificate rotation, and the operational realities of running TLS in production.

1. Why TLS for Kafka?

Without TLS, all Kafka traffic travels as plaintext across the network. Anyone with network access can read messages, inject records, or impersonate a broker.

TLS protects confidentiality (encryption) and identity (certificate verification). For authentication beyond identity, combine TLS with SASL (Day 44).

2. Key concepts — CA, keystore, truststore

CA (Certificate Authority)

Signs certificates. Any entity that trusts the CA will automatically trust any certificate the CA has signed. For development use a self-signed CA; for production use Let’s Encrypt or an internal corporate CA.

Keystore

Holds an entity’s own private key and signed certificate. The broker presents its keystore cert during the TLS handshake. Clients (in mTLS) present theirs too. Format: JKS (.jks) or PKCS12 (.p12).

Truststore

Holds trusted CA certificates. The client verifies the broker’s cert against the CA in its truststore. In mTLS the broker also has a truststore to verify client certs.

                    signs
CA key + CA cert ──────────▶ broker-signed.crt
                              (imported into broker keystore)

Client truststore              Broker keystore
┌─────────────┐                ┌──────────────────┐
│  ca-cert    │  verifies ◀─── │ broker-signed.crt│
│             │                │ broker private key│
└─────────────┘                └──────────────────┘

3. Generate certs with keytool

Step 1: Create a self-signed CA

openssl req -new -x509 \
  -keyout ca-key \
  -out ca-cert \
  -days 3650 \
  -subj "/CN=labs-ca" \
  -passout pass:ca-password

Step 2: Create broker keystore and CSR

# Generate broker key pair + self-signed cert in keystore
keytool -genkey \
  -alias broker \
  -keystore broker.keystore.jks \
  -keyalg RSA \
  -keysize 2048 \
  -dname "CN=broker,OU=labs,O=Boottech,L=Madrid,ST=Madrid,C=ES" \
  -storepass changeit \
  -keypass changeit \
  -validity 365

# Export certificate signing request (CSR)
keytool -certreq \
  -alias broker \
  -keystore broker.keystore.jks \
  -file broker.csr \
  -storepass changeit

This command as written will likely fail modern hostname verification — see §7. It sets only a CN (Common Name), with no Subject Alternative Name (SAN). Java 8u181+ and most current TLS clients no longer trust CN-only hostname matching by default; §7 shows the corrected version of this exact command.

Step 3: Sign CSR with CA and import into keystore

# Sign the broker CSR with the CA
openssl x509 -req \
  -CA ca-cert \
  -CAkey ca-key \
  -in broker.csr \
  -out broker-signed.crt \
  -days 365 \
  -CAcreateserial \
  -passin pass:ca-password

# Import CA cert into broker keystore (establishes chain of trust)
keytool -import \
  -alias CARoot \
  -file ca-cert \
  -keystore broker.keystore.jks \
  -storepass changeit \
  -noprompt

# Import signed broker cert into keystore
keytool -import \
  -alias broker \
  -file broker-signed.crt \
  -keystore broker.keystore.jks \
  -storepass changeit

Step 4: Create truststore (for clients and mTLS)

# Create truststore containing only the CA cert
keytool -import \
  -alias CARoot \
  -file ca-cert \
  -keystore kafka.truststore.jks \
  -storepass changeit \
  -noprompt

Step 5: Create client keystore (for mTLS)

# Same process as broker, with CN=labs-api
keytool -genkey -alias labs-api -keystore client.keystore.jks \
  -keyalg RSA -keysize 2048 -dname "CN=labs-api,OU=labs" \
  -storepass changeit -keypass changeit
keytool -certreq -alias labs-api -keystore client.keystore.jks \
  -file client.csr -storepass changeit
openssl x509 -req -CA ca-cert -CAkey ca-key -in client.csr \
  -out client-signed.crt -days 365 -CAcreateserial -passin pass:ca-password
keytool -import -alias CARoot -file ca-cert -keystore client.keystore.jks \
  -storepass changeit -noprompt
keytool -import -alias labs-api -file client-signed.crt \
  -keystore client.keystore.jks -storepass changeit

# Convert to PKCS12 for Spring Boot
keytool -importkeystore \
  -srckeystore client.keystore.jks \
  -destkeystore client.keystore.p12 \
  -deststoretype PKCS12 \
  -srcstorepass changeit \
  -deststorepass changeit \
  -noprompt

4. Broker SSL configuration

# server.properties (Kafka 4 KRaft mode)

# Dual listeners: PLAINTEXT for internal, SSL for external clients
listeners=PLAINTEXT://0.0.0.0:9092,SSL://0.0.0.0:9093
advertised.listeners=PLAINTEXT://broker:9092,SSL://broker:9093
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL

# Keystore — broker's own private key + signed certificate
ssl.keystore.location=/etc/kafka/secrets/broker.keystore.jks
ssl.keystore.password=changeit
ssl.key.password=changeit

# Truststore — CA cert to verify client certs (required for mTLS)
ssl.truststore.location=/etc/kafka/secrets/kafka.truststore.jks
ssl.truststore.password=changeit

# Client authentication: none | requested | required
ssl.client.auth=required

# TLS version
ssl.enabled.protocols=TLSv1.3,TLSv1.2

Docker Compose — mount secrets

connect:
  image: confluentinc/cp-kafka:7.6.0
  volumes:
    - ./secrets:/etc/kafka/secrets:ro
  environment:
    KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,SSL://0.0.0.0:9093
    KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092,SSL://broker:9093
    KAFKA_SSL_KEYSTORE_LOCATION: /etc/kafka/secrets/broker.keystore.jks
    KAFKA_SSL_KEYSTORE_PASSWORD: changeit
    KAFKA_SSL_KEY_PASSWORD: changeit
    KAFKA_SSL_TRUSTSTORE_LOCATION: /etc/kafka/secrets/kafka.truststore.jks
    KAFKA_SSL_TRUSTSTORE_PASSWORD: changeit
    KAFKA_SSL_CLIENT_AUTH: required

5. Spring Boot — application.yml SSL config

# application.yml — labs-api with Kafka SSL
spring:
  kafka:
    bootstrap-servers: broker:9093   # SSL port, not 9092
    security.protocol: SSL
    ssl:
      key-store: classpath:client.keystore.p12
      key-store-password: changeit
      key-store-type: PKCS12
      trust-store: classpath:kafka.truststore.jks
      trust-store-password: changeit
      trust-store-type: JKS

Place client.keystore.p12 and kafka.truststore.jks in src/main/resources/. In production, use environment variables or Spring Cloud Vault — never commit passwords to source control.

Environment variable override (recommended for prod)

spring:
  kafka:
    bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS:broker:9093}
    security.protocol: SSL
    ssl:
      key-store: ${KAFKA_SSL_KEYSTORE_PATH}
      key-store-password: ${KAFKA_SSL_KEYSTORE_PASSWORD}
      key-store-type: PKCS12
      trust-store: ${KAFKA_SSL_TRUSTSTORE_PATH}
      trust-store-password: ${KAFKA_SSL_TRUSTSTORE_PASSWORD}

Verify SSL connection

# Test SSL handshake from CLI
kafka-console-producer.sh \
  --bootstrap-server broker:9093 \
  --topic labs.events \
  --producer.config client-ssl.properties

# client-ssl.properties
# security.protocol=SSL
# ssl.truststore.location=kafka.truststore.jks
# ssl.truststore.password=changeit
# ssl.keystore.location=client.keystore.jks
# ssl.keystore.password=changeit

6. One-way TLS vs mTLS

For production Kafka clusters, always use mTLS. Combined with SASL (Day 44) you get both wire encryption and strong credential-based authentication.

7. SAN vs CN — the hostname verification gotcha in §3’s commands

Kafka clients validate the broker’s certificate hostname by default (ssl.endpoint.identification.algorithm=https, on since Kafka 2.0) — and modern Java/TLS stacks perform that check against the certificate’s Subject Alternative Name (SAN) extension, not the older CN field the keytool -genkey -dname "CN=broker,..." command in §3 relies on exclusively.

# Corrected version of §3 Step 2 — adds a SAN extension alongside the CN
keytool -genkey \
  -alias broker \
  -keystore broker.keystore.jks \
  -keyalg RSA \
  -keysize 2048 \
  -dname "CN=broker,OU=labs,O=Boottech,L=Madrid,ST=Madrid,C=ES" \
  -ext "SAN=DNS:broker,DNS:broker.labs.internal,DNS:localhost,IP:127.0.0.1" \
  -storepass changeit \
  -keypass changeit \
  -validity 365

# The CSR export must also carry the SAN extension through — openssl needs an extfile
openssl x509 -req -CA ca-cert -CAkey ca-key -in broker.csr \
  -out broker-signed.crt -days 365 -CAcreateserial -passin pass:ca-password \
  -extfile <(printf "subjectAltName=DNS:broker,DNS:broker.labs.internal,DNS:localhost,IP:127.0.0.1")

What breaks without this: a client connecting to broker:9093 performs hostname verification by comparing the connection hostname (broker) against the certificate’s SANs. A CN-only certificate as generated by §3’s original commands can fail this check on current JVM versions with a CertificateException: No subject alternative names present, even though the CA chain of trust is otherwise completely valid. List every hostname/IP a client might actually use to reach each broker (container name, internal DNS name, localhost for local testing) as SAN entries — and regenerate certs whenever a broker’s addressable hostnames change, not just when the cert expires.

The tempting but dangerous shortcut: setting ssl.endpoint.identification.algorithm= (empty) disables hostname verification entirely rather than fixing the certificate — this reduces mTLS to “any cert signed by a trusted CA is accepted regardless of which host presents it,” meaningfully weakening the security model. Fix the SAN, don’t disable the check.

8. Certificate expiration & rotation — the “365 days, then everything breaks” risk

Every cert in §3 is generated with -validity 365 / -days 365. Without a rotation plan, this is a scheduled full-cluster outage waiting to happen — TLS connections fail hard once a certificate expires, and unlike many other configuration problems, there’s no graceful degradation.

# Check remaining validity on a cert — script this into a recurring check, not a manual habit
keytool -list -v -keystore broker.keystore.jks -storepass changeit -alias broker | grep "until"

# Or via openssl, for scripting/monitoring integration
openssl x509 -enddate -noout -in broker-signed.crt

Rotation without downtime — the actual production pattern:

Generate the new certificate well before expiry (weeks, not days — leave room to catch problems).
Import the new cert into a new keystore alias, keeping the old one valid alongside it temporarily — Kafka can present either, and clients trusting the same CA accept both during the transition.
Roll brokers one at a time to pick up the new keystore (a rolling restart, same operational pattern as any other config change requiring a broker bounce).
Once all brokers and clients have transitioned, remove the old cert.

Why this deserves explicit planning, not just “renew when it’s close to expiring”: a cluster-wide simultaneous certificate expiry is one of the more painful categories of “everything was fine yesterday” outages — it’s entirely self-inflicted (you set the 365-day validity yourself) and entirely preventable with calendar-based monitoring (§10), yet it’s a genuinely common real-world incident because certificate expiry isn’t triggered by load or traffic, so it doesn’t show up in the metrics people usually watch.

9. Inter-broker listener configuration — missing from §4’s example

§4’s broker config sets up client-facing listeners but doesn’t address how brokers talk to each other — in a multi-broker cluster (unlike this single-broker dev setup), replication traffic (Day 10’s ISR fetch protocol) and controller communication also flow over a configured listener, which needs its own explicit designation.

# Which listener brokers use to talk to EACH OTHER (replication, controller comms)
inter.broker.listener.name=SSL

# If inter-broker traffic should use different creds/certs than client traffic,
# a dedicated internal listener is common in production:
listeners=CLIENT://0.0.0.0:9093,BROKER://0.0.0.0:9094,CONTROLLER://0.0.0.0:9095
listener.security.protocol.map=CLIENT:SSL,BROKER:SSL,CONTROLLER:SSL
inter.broker.listener.name=BROKER

Why this matters even though this capstone-style example runs one broker: the moment this setup scales beyond a single node (Day 8’s KRaft quorum, Day 10’s replication), inter.broker.listener.name determines whether broker-to-broker replication traffic is encrypted at all — leaving it unset/misconfigured can mean client traffic is properly secured over SSL while replication traffic between brokers silently falls back to plaintext, which defeats a meaningful part of the point of enabling TLS in the first place.

10. Performance overhead & monitoring

TLS isn’t free — the handshake and per-record encryption/decryption add real CPU cost, worth knowing about rather than discovering as an unexplained throughput regression after enabling it.

# Certificate expiry monitoring — the single most important TLS-specific check
# to wire into existing alerting (treat like disk space: check regularly, alert with lead time)
for cert in broker.keystore.jks client.keystore.jks; do
  expiry=$(keytool -list -v -keystore $cert -storepass changeit | grep "until" | head -1)
  echo "$cert: $expiry"
done

What to actually alert on: certificate expiry with enough lead time to rotate calmly (e.g. 30 days out), not a same-day emergency. This is exactly the kind of check that’s easy to skip because TLS “just works” for months at a time — until the day it silently doesn’t.

11. Common pitfalls

CN-only certificates without SAN entries — fails hostname verification on modern JVMs; every broker-reachable hostname/IP needs to be in the SAN, not just the CN (§7)
Disabling ssl.endpoint.identification.algorithm to work around a hostname mismatch — fixes the symptom by removing a real security check instead of fixing the underlying certificate (§7)
No certificate rotation plan before the 365-day validity period was chosen — a scheduled, entirely self-inflicted outage that’s easy to forget about because nothing signals it’s coming until it’s too late (§8)
Leaving inter.broker.listener.name unset in a multi-broker deployment — client traffic can be properly encrypted while replication traffic between brokers silently isn’t (§9)
Storing keystore/truststore passwords in plaintext .properties files committed to source control — the CLI verification example in §5 and the client-ssl.properties pattern are fine for local testing, but production needs the same credential-externalization discipline as Day 37 §8’s ConfigProvider pattern, applied to TLS passwords specifically

Key Takeaways

TLS encrypts all Kafka wire traffic — plaintext is readable by anyone on the network
CA signs certs — clients trust any cert signed by a CA in their truststore
Keystore holds YOUR private key + cert; truststore holds trusted CA certs
Add SSL listener on :9093 — keep PLAINTEXT on :9092 for internal cluster traffic
mTLS (ssl.client.auth=required) authenticates both sides — use in production
Certificates need Subject Alternative Names (SAN), not just CN — CN-only certs fail hostname verification on modern JVMs
Plan certificate rotation before generating certs, not after — expiry is a scheduled, entirely preventable outage if monitored with lead time
inter.broker.listener.name controls whether broker-to-broker replication traffic is actually encrypted — don’t assume client-facing SSL config covers it
Spring Boot: security.protocol=SSL + ssl.* props in application.yml

Support me through GitHub Sponsors.

Thank you for Reading !! See you in the next post.

➡️ Day 44: SASL auth — PLAIN, SCRAM-SHA, OAuth 2.0

Resources

📘 Kafka: The Definitive Guide — Chapter 12 (Security)
🌐 kafka.apache.org/documentation/#security_ssl
🌐 Kafka — inter-broker listener configuration

Link to Medium blog