60-Day Kafka 4 Learning Plan · Week 7 — Day 49 of 60

---

60-Day Kafka 4 Learning Plan · Week 7 Capstone — Security & Monitoring
Sources: Kafka: The Definitive Guide Ch.10, 12 · kafka.apache.org/documentation/#security

Goal

Wire everything from Week 7 into a single Docker Compose stack: Kafka 4 KRaft with TLS + SASL_SSL + ACLs, labs-api producing, labs-socket consuming, and a full Prometheus + Grafana monitoring pipeline.

What we’re building

labs-api ──WRITE──► Kafka Broker (KRaft, 1 node) ──READ──► labs-socket
(SCRAM-SHA-512       ┌──────────────────────┐              (labs.* prefix
 SASL_SSL :9094)     │ SSL      :9093  mTLS  │              ACL, READ +
                     │ SASL_SSL :9094        │              Describe)
                     │ ACLs: StandardAuthorizer│
                     │ JMX      :9999 (debug) │
                     └──────────────────────┘
                              │ (internal PLAINTEXT :9092,
                              │  not published to host)
                              ▼
                     kafka-bootstrap (one-shot: users, topics, ACLs)
                     kafka-exporter :9308
                              │
                     Prometheus :9090 ──alerts──► (Slack, from Week 7 Day 48)
                              │
                     Grafana :3000

Five services beyond the broker: kafka-bootstrap (one-shot provisioning), labs-api, labs-socket, kafka-exporter, prometheus, grafana. Compose’s depends_on: condition: service_completed_successfully enforces the order — nothing that needs a SCRAM credential or an ACL starts before kafka-bootstrap has created it.

1. Project structure

kafka-secure-cluster-capstone-demo/
├── docker-compose.yml
├── .env.example                   # copy to .env — never commit the real one
├── secrets/
│   └── generate-certs.sh          # produces keystore, truststore, and credential files
├── config/
│   ├── prometheus.yml
│   ├── alert_rules.yml
│   ├── dashboards/kafka-secure-cluster.json
│   └── provisioning/
│       ├── datasources/prometheus.yml
│       └── dashboards/dashboard.yml
├── scripts/
│   └── bootstrap.sh               # creates SCRAM users, topics, ACLs
├── client/
│   └── index.html                 # live WebSocket feed, unchanged from Week 3
├── labs-api/
│   ├── Dockerfile
│   ├── pom.xml
│   └── src/main/...
└── labs-socket/
    ├── Dockerfile
    ├── pom.xml
    └── src/main/...

2. Fixing the broker image

The source material specified confluentinc/cp-kafka:7.6.0. Confluent Platform’s version numbers don’t track Kafka’s — CP 7.6 ships Kafka 3.6, and Kafka 3.6 still supports ZooKeeper mode. For an article titled “Kafka 4 KRaft,” that’s the wrong image. This series has been using the official apache/kafka:4.0.0 image since Week 1 — Kafka 4.0 dropped ZooKeeper support entirely, so KRaft isn’t a mode you opt into, it’s the only mode there is.

The official image uses the same KAFKA_-prefixed environment variable convention Confluent’s image made popular, so the config translates directly. One thing it does not do: Confluent’s entrypoint auto-creates a SASL/SCRAM user from whatever JAAS config you hand it at container start. The upstream Apache image doesn’t have that convenience script — the “admin” SCRAM user has to be created explicitly, which is why kafka-bootstrap exists as its own compose service instead of being folded into a broker startup hook.

3. The listener topology

Four listeners on one broker:

PLAINTEXT   :9092   internal only, no host port published
SSL         :9093   mTLS — client certificate required
SASL_SSL    :9094   SCRAM-SHA-512 — no client certificate required
CONTROLLER  :29093  KRaft quorum traffic, internal only

The two design decisions worth explaining:

Why PLAINTEXT stays alive at all. Creating a SCRAM user requires authenticating as an existing SCRAM user — a bootstrapping problem. Rather than invent a workaround, kafka-bootstrap uses the unauthenticated internal listener to seed the first credentials (admin, labs-api, labs-socket), then switches to SASL_SSL on :9094 for everything ACL-related, once admin actually has a password. kafka-exporter also uses this listener, since its job is reading public cluster metadata, not moving data. It works because the listener is never bound to a host port — docker-compose.yml publishes 9093, 9094, and 9999, not 9092. Anything on the host network cannot reach it; only containers on the compose network can.

Why SASL_SSL doesn’t also require a client certificate. ssl.client.auth is a broker-wide default, but Kafka lets you override it per listener with listener.name.<name>.ssl.client.auth. The compose file sets it to required on SSL and none on SASL_SSL:

KAFKA_LISTENER_NAME_SSL_SSL_CLIENT_AUTH: required
KAFKA_LISTENER_NAME_SASL_SSL_SSL_CLIENT_AUTH: none

Without that override, the global default from the TLS listener leaks onto the SASL_SSL listener too, and both labs-api and labs-socket — which only carry a truststore, not a client keystore — fail the TLS handshake before SASL is even attempted. The failure mode is a generic SSLHandshakeException, not an auth error, which makes it a slow one to debug the first time you hit it.

4. ANONYMOUS as super user: not the bug, the host port was

The source draft configured:

KAFKA_SUPER_USERS: "User:admin;User:ANONYMOUS"

On a listener security-protocol map where PLAINTEXT resolves every connection to the principal User:ANONYMOUS, granting ANONYMOUS super-user status turns any reachable PLAINTEXT connection into unrestricted, unauthenticated cluster admin — full read/write/alter-config on every topic, bypassing every ACL just written. The first instinct, and the one this article shipped with initially, is to delete User:ANONYMOUS from KAFKA_SUPER_USERS outright. Running the stack proved that instinct wrong.

KRaft’s own broker↔controller registration traffic rides the unauthenticated CONTROLLER listener — PLAINTEXT there too, so also principal ANONYMOUS — and StandardAuthorizer gates BROKER_REGISTRATION and CONTROLLER_REGISTRATION behind a ClusterAction check. Strip ANONYMOUS‘s super-user status and the broker can no longer register with its own single-node controller quorum:

org.apache.kafka.common.errors.ClusterAuthorizationException: Request ...
is not authorized.
[BrokerLifecycleManager id=1] Unable to register broker 1 because the
controller returned error CLUSTER_AUTHORIZATION_FAILED

The broker never comes up. So the real question isn’t “should ANONYMOUS be a super user” — it’s “which listeners resolve to ANONYMOUS, and can anything outside the trust boundary reach them.” PLAINTEXT (:9092) and CONTROLLER (:29093) both resolve to ANONYMOUS, and neither one gets a ports: entry in docker-compose.yml — they exist only on the compose network, reachable by kafka-bootstrap and the broker’s own controller quorum, never by the host. That’s what actually makes the grant safe: not the absence of the principal from super.users, but the fact that nothing outside the container network can ever present that principal. Keeping User:ANONYMOUS in KAFKA_SUPER_USERS and never publishing those two ports does what deleting the grant entirely cannot — it lets the cluster start.

The lesson generalizes past this one variable: a security control checked in isolation (“is ANONYMOUS a super user? no.”) can look correct while the system it’s protecting no longer functions, or — worse, if the mistake runs the other way — while a different control silently compensates for it. Test the actual failure mode, not the config line that looks like the failure mode.

5. Provisioning: scripts/bootstrap.sh

#!/bin/bash
set -euo pipefail

BIN=/opt/kafka/bin
BS_PLAINTEXT="broker:9092"
BS_SASL="broker:9094"
ADMIN_PROPS="/tmp/admin.properties"

cat > "$ADMIN_PROPS" <<EOF
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="${KAFKA_ADMIN_PASSWORD}";
ssl.truststore.location=/etc/kafka/secrets/kafka.truststore.jks
ssl.truststore.password=${SSL_TRUSTSTORE_PASSWORD}
EOF

declare -A USERS=(
  [admin]="${KAFKA_ADMIN_PASSWORD}"
  [labs-api]="${LABS_API_PASSWORD}"
  [labs-socket]="${LABS_SOCKET_PASSWORD}"
)
for name in "${!USERS[@]}"; do
  "$BIN/kafka-configs.sh" --bootstrap-server "$BS_PLAINTEXT" --alter \
    --add-config "SCRAM-SHA-512=[iterations=8192,password=${USERS[$name]}]" \
    --entity-type users --entity-name "$name"
done

"$BIN/kafka-topics.sh" --bootstrap-server "$BS_PLAINTEXT" \
  --create --if-not-exists --topic labs.events --partitions 3 --replication-factor 1
"$BIN/kafka-topics.sh" --bootstrap-server "$BS_PLAINTEXT" \
  --create --if-not-exists --topic labs.events.DLT --partitions 1 --replication-factor 1

"$BIN/kafka-acls.sh" --bootstrap-server "$BS_SASL" --command-config "$ADMIN_PROPS" \
  --add --allow-principal User:labs-api \
  --operation Write --operation Describe --topic labs.events

"$BIN/kafka-acls.sh" --bootstrap-server "$BS_SASL" --command-config "$ADMIN_PROPS" \
  --add --allow-principal User:labs-socket \
  --operation Read --operation Describe \
  --topic labs. --resource-pattern-type prefixed

"$BIN/kafka-acls.sh" --bootstrap-server "$BS_SASL" --command-config "$ADMIN_PROPS" \
  --add --allow-principal User:labs-socket \
  --operation Read --group labs-socket-group

"$BIN/kafka-acls.sh" --bootstrap-server "$BS_SASL" --command-config "$ADMIN_PROPS" \
  --add --allow-principal User:labs-socket \
  --operation Write --topic labs.events.DLT

Full listing including topic pre-creation and the closing --list audit is in the demo repo. Two ACL corrections from the source draft, both found by tracing what each service actually does at runtime rather than what the checklist assumed:

labs-api doesn’t need a consumer-group ACL. The original bootstrap script granted labs-api Read on group labs-api-group — but labs-api is a producer; it never calls .subscribe(), never joins a consumer group. That ACL was dead privilege, most likely copy-pasted from a template written for a service that does both. Least-privilege ACL sets should be derived from what the code does, not assumed from the pattern.

labs-socket needs a separate, narrower ACL to write its own dead-letter topic. labs-socket‘s DefaultErrorHandler publishes to labs.events.DLT after retries are exhausted (see KafkaConsumerConfig.java below) — that’s a Write operation on a specific topic. The prefixed Read/Describe ACL on labs. does not grant Write on anything. Without the explicit exact-match ACL on labs.events.DLT, dead-letter publishing fails with TopicAuthorizationException, and because it fails inside the error handler’s own recovery path, the original message’s offset still gets committed — the failed record is gone, not retried, not in the DLT, just silently dropped. This is the kind of bug that only shows up under load, during an incident, which is the worst possible time to discover it.

6. TLS material: secrets/generate-certs.sh

The source material’s file tree listed broker.keystore.jks, kafka.truststore.jks, and client.keystore.p12 as pre-existing binary files. Binaries can’t ship in a blog post, and client.keystore.p12 turns out to be unused by anything in this stack anyway — since SASL_SSL has ssl.client.auth=none (§3), neither labs-api nor labs-socket needs a client certificate, only a truststore to verify the broker’s. What ships instead is the script that generates both real files reproducibly:

openssl req -new -x509 -keyout ca.key -out ca.crt -days "$VALID_DAYS" \
  -subj "/CN=labs-kafka-dev-ca/OU=labs/O=boottechsolutions/L=Paris/C=FR" \
  -passout pass:"$SSL_KEYSTORE_PASSWORD"

keytool -genkeypair -alias broker -keyalg RSA -keysize 2048 -validity "$VALID_DAYS" \
  -keystore broker.keystore.jks -storetype JKS -storepass "$SSL_KEYSTORE_PASSWORD" \
  -keypass "$SSL_KEY_PASSWORD" -dname "$KEYTOOL_DNAME" \
  -ext "SAN=DNS:broker,DNS:localhost,IP:127.0.0.1"

Two details here cost real debugging time and are worth calling out explicitly:

-dname and -subj do not speak the same syntax. openssl -subj wants the leading-slash form (/CN=x/OU=y); keytool -dname wants comma-separated RFC 2253 (CN=x, OU=y). Pass the slash form to keytool and it fails with keytool error: java.io.IOException: Invalid keyword "/CN" — an error that gives no hint the fix is just reformatting the same string. The script keeps two variables, OPENSSL_SUBJ-style inline strings for the openssl calls and $KEYTOOL_DNAME for the keytool ones, precisely so this doesn’t get merged into one “DRY” variable that’s subtly wrong for half its uses.

-storetype JKS is not optional, despite the .jks filename. keytool has defaulted to PKCS12 since JDK 9. Omit -storetype JKS and you get a file named broker.keystore.jks that is actually PKCS12 — and PKCS12 keystores reject a -keypass distinct from -storepass, silently downgrading to a single shared password with only a warning, which breaks KAFKA_SSL_KEY_PASSWORD being intentionally different from KAFKA_SSL_KEYSTORE_PASSWORD later in docker-compose.yml.

The SAN=DNS:broker extension matters more than it looks: broker is the compose service’s hostname, and KAFKA_ADVERTISED_LISTENERS tells clients to connect to broker:9094. If the certificate’s Subject Alternative Name doesn’t include that exact hostname, every client-side TLS handshake fails hostname verification even though the CA chain is perfectly valid — a mismatch between two correct-looking pieces of config that only shows up at connection time.

The script also writes three plaintext credential files — broker_keystore_creds, broker_key_creds, broker_truststore_creds — that have nothing to do with openssl or keytool at all. That requirement comes from the broker side, covered next. Full script — CSR, CA signing, importing into both stores, idempotent cleanup on rerun — is in the repo; run it once after copying .env.example to .env.

7. Wiring docker-compose.yml

services:
  broker:
    image: apache/kafka:4.0.0
    hostname: broker
    environment:
      CLUSTER_ID: ${CLUSTER_ID}
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@broker:29093
      KAFKA_LOG_DIRS: /var/lib/kafka/data

      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,SSL://0.0.0.0:9093,SASL_SSL://0.0.0.0:9094,CONTROLLER://0.0.0.0:29093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092,SSL://broker:9093,SASL_SSL://broker:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_SSL:SASL_SSL,CONTROLLER:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: SASL_SSL
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER

      KAFKA_SASL_ENABLED_MECHANISMS: SCRAM-SHA-512
      KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: SCRAM-SHA-512
      # Double underscore, not single — see the env-var gotcha below.
      KAFKA_LISTENER_NAME_SASL__SSL_SCRAM___SHA___512_SASL_JAAS_CONFIG: >-
        org.apache.kafka.common.security.scram.ScramLoginModule required
        username="admin" password="${KAFKA_ADMIN_PASSWORD}";
      # Mandatory the moment any SASL_* listener is advertised — see below.
      KAFKA_OPTS: -Djava.net.preferIPv4Stack=true

      # Filename + credentials-file convention — see below, not LOCATION/PASSWORD.
      KAFKA_SSL_KEYSTORE_FILENAME: broker.keystore.jks
      KAFKA_SSL_KEY_CREDENTIALS: broker_key_creds
      KAFKA_SSL_KEYSTORE_CREDENTIALS: broker_keystore_creds
      KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.truststore.jks
      KAFKA_SSL_TRUSTSTORE_CREDENTIALS: broker_truststore_creds
      KAFKA_SSL_CLIENT_AUTH: required
      KAFKA_LISTENER_NAME_SASL__SSL_SSL_CLIENT_AUTH: none

      KAFKA_AUTHORIZER_CLASS_NAME: org.apache.kafka.metadata.authorizer.StandardAuthorizer
      KAFKA_SUPER_USERS: "User:admin;User:ANONYMOUS"
      KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "false"
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"

      KAFKA_JMX_PORT: 9999
      KAFKA_JMX_HOSTNAME: broker
      KAFKA_JMX_OPTS: >-
        -Dcom.sun.management.jmxremote
        -Dcom.sun.management.jmxremote.authenticate=false
        -Dcom.sun.management.jmxremote.ssl=false
        -Djava.rmi.server.hostname=broker
        -Dcom.sun.management.jmxremote.rmi.port=9999

      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
    volumes:
      - ./secrets:/etc/kafka/secrets:ro
      - kafka_data:/var/lib/kafka/data
    ports:
      - "9093:9093"
      - "9094:9094"
      - "9999:9999"
    healthcheck:
      test: ["CMD-SHELL", "/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server broker:9092"]
      interval: 10s
      timeout: 10s
      retries: 15
      start_period: 30s

  kafka-bootstrap:
    image: apache/kafka:4.0.0
    entrypoint: ["/bin/bash", "/scripts/bootstrap.sh"]
    environment:
      KAFKA_ADMIN_PASSWORD: ${KAFKA_ADMIN_PASSWORD}
      LABS_API_PASSWORD: ${LABS_API_PASSWORD}
      LABS_SOCKET_PASSWORD: ${LABS_SOCKET_PASSWORD}
      SSL_TRUSTSTORE_PASSWORD: ${SSL_TRUSTSTORE_PASSWORD}
    volumes:
      - ./scripts/bootstrap.sh:/scripts/bootstrap.sh:ro
      - ./secrets:/etc/kafka/secrets:ro
    depends_on:
      broker:
        condition: service_healthy

  labs-api:
    build: { context: ./labs-api }
    ports: ["8080:8080"]
    environment:
      KAFKA_BOOTSTRAP_SERVERS: broker:9094
      KAFKA_SASL_USERNAME: labs-api
      KAFKA_SASL_PASSWORD: ${LABS_API_PASSWORD}
      SSL_TRUSTSTORE_LOCATION: /app/kafka.truststore.jks
      SSL_TRUSTSTORE_PASSWORD: ${SSL_TRUSTSTORE_PASSWORD}
    volumes:
      - ./secrets/kafka.truststore.jks:/app/kafka.truststore.jks:ro
    depends_on:
      broker: { condition: service_healthy }
      kafka-bootstrap: { condition: service_completed_successfully }

  labs-socket:
    build: { context: ./labs-socket }
    ports: ["8081:8081"]
    environment:
      KAFKA_BOOTSTRAP_SERVERS: broker:9094
      KAFKA_SASL_USERNAME: labs-socket
      KAFKA_SASL_PASSWORD: ${LABS_SOCKET_PASSWORD}
      SSL_TRUSTSTORE_LOCATION: /app/kafka.truststore.jks
      SSL_TRUSTSTORE_PASSWORD: ${SSL_TRUSTSTORE_PASSWORD}
    volumes:
      - ./secrets/kafka.truststore.jks:/app/kafka.truststore.jks:ro
    depends_on:
      broker: { condition: service_healthy }
      kafka-bootstrap: { condition: service_completed_successfully }

  kafka-exporter:
    image: danielqsj/kafka-exporter:latest
    command: ["--kafka.server=broker:9092", "--web.listen-address=:9308"]
    ports: ["9308:9308"]
    depends_on:
      broker: { condition: service_healthy }

  prometheus:
    image: prom/prometheus:latest
    ports: ["9090:9090"]
    volumes:
      - ./config/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./config/alert_rules.yml:/etc/prometheus/alert_rules.yml:ro
      - prometheus_data:/prometheus
    depends_on: [kafka-exporter, labs-api, labs-socket]

  grafana:
    image: grafana/grafana:latest
    ports: ["3000:3000"]
    environment:
      GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PASSWORD}
    volumes:
      - ./config/provisioning:/etc/grafana/provisioning:ro
      - ./config/dashboards:/var/lib/grafana/dashboards:ro
      - grafana_data:/var/lib/grafana
    depends_on: [prometheus]

volumes:
  kafka_data:
  prometheus_data:
  grafana_data:

One more fix worth flagging: KAFKA_LOG_DIRS is set explicitly to /var/lib/kafka/data, matching the mounted kafka_data volume. The base image’s default log directory doesn’t line up with that path — without this line, the volume mount is present but the broker never writes to it, and a docker compose down && up silently starts from an empty log on every restart. The bug hides well because nothing errors; the cluster just forgets its data.

8. Spring Boot: labs-api/application.yml

spring:
  application:
    name: labs-api
  kafka:
    bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS:localhost:9094}
    security:
      protocol: SASL_SSL
    properties:
      sasl:
        mechanism: SCRAM-SHA-512
        jaas:
          config: >-
            org.apache.kafka.common.security.scram.ScramLoginModule required
            username="${KAFKA_SASL_USERNAME:labs-api}"
            password="${KAFKA_SASL_PASSWORD:labs-api-secret}";
      ssl:
        truststore:
          location: ${SSL_TRUSTSTORE_LOCATION:secrets/kafka.truststore.jks}
          password: ${SSL_TRUSTSTORE_PASSWORD:changeit}
    producer:
      client-id: labs-api-producer

kafka:
  topic:
    labs-events: labs.events

management:
  endpoints:
    web:
      exposure:
        include: health,prometheus
  metrics:
    tags:
      application: labs-api

labs-socket/application.yml mirrors this with its own credential and adds consumer.group-id: labs-socket-group plus the DLT-capable producer block — full listing in the repo.

Notice what’s not here: a TopicConfig bean creating NewTopics at startup. The Week 3 version of labs-api had one — Spring Kafka’s KafkaAdmin auto-creates any NewTopic bean it finds, which is convenient until ACLs are involved. labs-api‘s ACL grants Write and Describe on labs.events, not Create on the cluster. Leaving the TopicConfig bean in means the app tries to create a topic it isn’t authorized to create on every startup, logs a swallowed TopicAuthorizationException from KafkaAdmin, and only happens to work because kafka-bootstrap already created the topic first — a race that’s currently won by depends_on, but is one compose refactor away from breaking. Removing the bean and relying on kafka-bootstrap as the single source of topic truth makes the dependency explicit instead of accidental.

§9 Monitoring: Prometheus + Grafana

config/prometheus.yml scrapes three targets: kafka-exporter:9308 for cluster-level metrics (broker up, produce rate, consumer lag), plus labs-api:8080/actuator/prometheus and labs-socket:8081/actuator/prometheus for the application layer — request rates, JVM memory, HTTP status distribution — via micrometer-registry-prometheus on the classpath.

scrape_configs:
  - job_name: kafka
    static_configs: [{ targets: ["kafka-exporter:9308"] }]
  - job_name: labs-api
    metrics_path: /actuator/prometheus
    static_configs: [{ targets: ["labs-api:8080"] }]
  - job_name: labs-socket
    metrics_path: /actuator/prometheus
    static_configs: [{ targets: ["labs-socket:8081"] }]

JMX and Prometheus are two separate pipelines, and that’s intentional, not an oversight. KAFKA_JMX_PORT: 9999 (§7) exposes JMX over RMI, for jconsole or VisualVM ad-hoc inspection — Prometheus cannot scrape RMI directly; it needs an HTTP endpoint. kafka-exporter doesn’t bridge JMX either — it talks the Kafka protocol itself (AdminClient + consumer-group APIs) against broker:9092, which is why it needs no JMX config at all. If your cluster needs the MBeans JMX exposes that kafka-exporter doesn’t cover (GC pause time, request-handler idle ratio), the correct addition is the Prometheus JMX Exporter as a Java agent on the broker’s KAFKA_OPTS, not pointing Prometheus at :9999 and expecting it to work — a mistake common enough that it’s worth stating plainly here.

config/alert_rules.yml ships four rules — broker down, consumer lag over 1000 for 5 minutes, zero produce throughput for 10 minutes, and each Spring service’s actuator going unreachable:

- alert: KafkaConsumerGroupLagHigh
  expr: sum(kafka_consumergroup_lag{consumergroup="labs-socket-group"}) > 1000
  for: 5m
  labels: { severity: warning }
  annotations:
    summary: "labs-socket-group is falling behind"

The Slack routing and the six-panel dashboard layout were built in Week 7 Day 48 — config/dashboards/kafka-secure-cluster.json here is a compact five-panel starter (broker up, produce rate, consumer lag, HTTP request rate, under-replicated partitions) wired through Grafana’s file-based provisioning so it’s live the moment docker compose up finishes, rather than something you click through manually.

Testing

1. Configure secrets

cp .env.example .env
# edit .env with real passwords

2. Generate TLS material

set -a; source .env; set +a
./secrets/generate-certs.sh

This writes secrets/broker.keystore.jks and secrets/kafka.truststore.jks. Both are gitignored — regenerate them locally, never commit them.

3. Bring the stack up

docker compose up -d --build

Startup order, enforced by depends_on:

1. broker — starts and passes its healthcheck (internal PLAINTEXT listener).
2. kafka-bootstrap — seeds SCRAM users, creates labs.events / labs.events.DLT, and applies ACLs, then exits 0.
3. labs-api, labs-socket, kafka-exporter — start once kafka-bootstrap has completed successfully.
4. prometheus, grafana — start once the app services are up.

4. Verify

# publish an event
curl -s -X POST http://localhost:8080/api/events \
  -H "Content-Type: application/json" \
  -d '{"type":"ORDER_CREATED","payload":"first secure order"}'

# open the live feed
open client/index.html   # or just double-click it

# Prometheus targets
open http://localhost:9090/targets

# Grafana (admin / $GRAFANA_PASSWORD)
open http://localhost:3000

5. Prove the ACLs are real

# labs-socket has no Write ACL on labs.events — this must fail with
# TopicAuthorizationException:
docker compose exec broker /opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server broker:9094 \
  --topic labs.events \
  --producer.config /dev/stdin <<EOF
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="labs-socket" password="$LABS_SOCKET_PASSWORD";
ssl.truststore.location=/etc/kafka/secrets/kafka.truststore.jks
ssl.truststore.password=$SSL_TRUSTSTORE_PASSWORD
EOF

Ports

Integration Testing

EmbeddedKafka starts an in-process plaintext broker — it doesn’t support SASL_SSL. Rather than skip integration tests or fight the harness, both services carry a test Spring profile (application-test.yml) that overrides security.protocol back to PLAINTEXT for the embedded broker only:

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@ActiveProfiles("test")
@EmbeddedKafka(partitions = 3, topics = {"labs.events", "labs.events.DLT"})
@DirtiesContext
class EventControllerIntegrationTest {
    // ...
}

This validates the producer/consumer wiring, the DLT recovery path, and the HTTP contract — it does not exercise SASL negotiation, TLS handshakes, or ACL enforcement. Those three are the actual subject of this article, and EmbeddedKafka structurally cannot test them. Verify those against the real stack instead, either manually (the README’s §5 shows labs-socket attempting an unauthorized produce and getting rejected) or by scripting the same kafka-console-producer.sh / kafka-console-consumer.sh calls into CI as a smoke test after docker compose up. Treat “SASL/TLS/ACL config is correct” and “application logic is correct” as two different test suites, because they require two different kinds of broker.

Performance considerations

TLS and SASL both add CPU cost on every connection — the TLS handshake and SCRAM’s iterated HMAC (iterations=8192 here) are the expensive parts, not steady-state throughput, since Kafka connections are long-lived and reused. For a producer or consumer that reconnects frequently (short-lived serverless functions, autoscaling pods with aggressive churn), that handshake cost is paid far more often and is worth benchmarking before assuming the security layer is “free.” StandardAuthorizer‘s ACL check is an in-memory lookup against the metadata cache per request — negligible next to the network round-trip it gates. The real cost of ACLs is operational, not computational: every new topic or consumer group is a provisioning step, not a config-free kafka-topics.sh --create.

Common pitfalls

• A super-user grant is only as dangerous as the listeners that resolve to it. Deleting User:ANONYMOUS from KAFKA_SUPER_USERS looks like the obvious fix and breaks KRaft’s own controller registration instead. Audit which listeners map to which principal and whether those listeners have a host port, not just which principals are super users (§4).
• Per-listener ssl.client.auth not set. Global ssl.client.auth silently applies to every TLS-based listener, including SASL_SSL, producing a confusing SSLHandshakeException instead of the expected auth failure (§3).
• keytool -dname and openssl -subj don’t share a syntax. The leading-slash DN form works for openssl, and fails on keytool with Invalid keyword "/CN" — an error that doesn’t point at the actual problem, which is formatting, not content (§6).
• keytool‘s default keystore type is PKCS12, not JKS, since JDK 9. A .jks filename doesn’t make it one; without -storetype JKS, a distinct -keypass is silently dropped with only a warning, and the file isn’t the format Kafka’s config claims it is (§6).
• The official apache/kafka image needs a keystore file, not a password env var. KAFKA_SSL_KEYSTORE_LOCATION/_PASSWORD — the Confluent cp-kafka convention — are silently ignored. The mandatory KAFKA_SSL_KEYSTORE_FILENAME + _CREDENTIALS pair fails with an opaque !1: unbound variable if skipped, nothing pointing at SSL config at all (§7).
• KAFKA_OPTS must be set the instant any SASL_* listener exists — unconditionally, regardless of which SASL config style is in use. Same opaque unbound-variable failure if it’s missing (§7).
• A single underscore in an env var can silently target the wrong property. KAFKA_LISTENER_NAME_SASL_SSL_... (single underscores) converts to a property Kafka never reads; the fix needs SASL__SSL (double) to preserve the listener name’s literal underscore. The broker starts fine and only fails once a client tries to authenticate, with an error that looks like a client-side JAAS bug (§7).
• A relative truststore path resolves against the container’s working directory, not the mount point. labs-api‘s lazy producer hid this through a clean startup; labs-socket‘s eager @KafkaListener container caught it immediately with NoSuchFileException (§7).
• Consumer publishing to a DLT it has no Write ACL for. Fails inside the error handler’s own recovery path, which means the original message’s offset still commits — a message is lost with no DLT record and no loud error, only a log line easy to miss under load (§5).
• NewTopic beans racing pre-provisioned ACLs. KafkaAdmin‘s auto-create tries to run on every app startup, needs Create on the cluster, and works only because it currently loses the race to kafka-bootstrap — until a compose change reorders startup (§8).
• JMX port exposed, assumed Prometheus-scrapable. RMI isn’t HTTP; kafka-exporter doesn’t touch JMX at all. Two independent metrics pipelines that happen to share a checklist item (§9).
• Certificate SAN missing the Docker service hostname. A textbook-correct CA chain still fails hostname verification if broker (the compose service name clients actually connect to) isn’t in the cert’s SAN list (§6).

Key Takeaways

• TLS, SASL, and ACLs are three independent controls, not stages of a pipeline — each is fully bypassable on its own if an unauthenticated listener with super-user rights is reachable from outside the trust boundary. The control that actually matters is which listeners can be reached from where, not which principals are super users in isolation.
• Least-privilege ACLs come from tracing what each service actually does at runtime (does it consume? does it write to a DLT?), not from templating one service’s ACL set onto another.
• EmbeddedKafka cannot validate SASL/TLS/ACL wiring — that verification has to happen against the real secured broker, as a separate, deliberate test step.

Week 7 security checklist

---

The complete source code is available on GitHub.

Support me through GitHub Sponsors.

Thank you for Reading !! See you in the next post.

Next

➡️ Week 8, Day 50: Kafka 4 on Kubernetes — Strimzi KRaft mode, StatefulSets

References

• Kafka Security Documentation
• KIP-500 / KRaft Overview
• KIP-768: SASL/OAUTHBEARER
• Prometheus JMX Exporter
• danielqsj/kafka_exporter
• Base project: spring-kafka-websocket-pipeline-demo

Link to Medium blog