Kafka

60-Day Kafka 4 Learning Plan · Week 7 — Day 44 of 60

Editorial Team

60-Day Kafka 4 Learning Plan · Week 7 — Security & Monitoring Sources: Kafka: The Definitive Guide Ch.12 · kafka.apache.org/documentation/#security_sasl

Goal

Add credential-based authentication to Kafka using SASL. Understand the difference between TLS (encryption) and SASL (authentication), configure SASL/PLAIN for development, SASL/SCRAM-SHA-512 for production, solve the KRaft SCRAM bootstrap chicken-and-egg problem, understand why authentication alone isn’t authorization, and wire Spring Boot (labs-api) to connect with SASL_SSL.

1. TLS vs SASL — encryption vs authentication

Combined protocol: SASL_SSL = SASL authentication over TLS encryption. This is the correct production setting. SASL_PLAINTEXT sends credentials unencrypted — never use in production.

2. SASL/PLAIN — username + password

The simplest SASL mechanism. Credentials are defined in the broker’s JAAS config file. No dynamic user management — adding a user requires editing the file and restarting the broker.

⚠ Always use SASL/PLAIN over TLS (SASL_SSL), never over SASL_PLAINTEXT.

kafka_server_jaas.conf

KafkaServer {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="admin"
  password="admin-secret"
  user_admin="admin-secret"
  user_labs-api="labs-api-secret"
  user_labs-socket="labs-socket-secret";
};

server.properties additions

listeners=PLAINTEXT://0.0.0.0:9092,SASL_SSL://0.0.0.0:9094
sasl.enabled.mechanisms=PLAIN
sasl.mechanism.inter.broker.protocol=PLAIN
inter.broker.listener.name=SASL_SSL

Start broker with JAAS config

export KAFKA_OPTS="-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf"
# or set as Docker env var:
# KAFKA_OPTS: -Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf

3. SASL/SCRAM-SHA-512 — hashed credentials ★

SCRAM stores credentials hashed inside Kafka itself (in the @metadata internal topic in KRaft). No JAAS file edits needed for new users — add, modify, and delete users at runtime via kafka-configs.sh. This is the recommended production mechanism.

Create SCRAM users at runtime

# Create admin user
kafka-configs.sh \
  --bootstrap-server broker:9092 \
  --alter \
  --add-config 'SCRAM-SHA-512=[iterations=8192,password=admin-secret]' \
  --entity-type users \
  --entity-name admin

# Create labs-api service account
kafka-configs.sh \
  --bootstrap-server broker:9092 \
  --alter \
  --add-config 'SCRAM-SHA-512=[iterations=8192,password=labs-api-secret]' \
  --entity-type users \
  --entity-name labs-api

# Create labs-socket service account
kafka-configs.sh \
  --bootstrap-server broker:9092 \
  --alter \
  --add-config 'SCRAM-SHA-512=[iterations=8192,password=labs-socket-secret]' \
  --entity-type users \
  --entity-name labs-socket

# List all users
kafka-configs.sh --bootstrap-server broker:9092 --describe --entity-type users

# Delete a user
kafka-configs.sh \
  --bootstrap-server broker:9092 \
  --alter \
  --delete-config 'SCRAM-SHA-512' \
  --entity-type users \
  --entity-name old-service

kafka_server_jaas.conf — SCRAM (broker credentials only)

KafkaServer {
  org.apache.kafka.common.security.scram.ScramLoginModule required
  username="admin"
  password="admin-secret";
};

The broker only needs its own credentials in JAAS. All other users are stored in Kafka.

4. Broker server.properties — SASL_SSL listener

# Listeners: keep PLAINTEXT for internal, add SASL_SSL for authenticated clients
listeners=PLAINTEXT://0.0.0.0:9092,SSL://0.0.0.0:9093,SASL_SSL://0.0.0.0:9094
advertised.listeners=PLAINTEXT://broker:9092,SSL://broker:9093,SASL_SSL://broker:9094
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_SSL:SASL_SSL

# SASL mechanisms
sasl.enabled.mechanisms=SCRAM-SHA-512
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
inter.broker.listener.name=SASL_SSL

# Inline JAAS for the SASL_SSL listener (alternative to external JAAS file)
listener.name.sasl_ssl.scram-sha-512.sasl.jaas.config=\
  org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="admin" \
  password="admin-secret";

# SSL settings (from Day 43)
ssl.keystore.location=/etc/kafka/secrets/broker.keystore.jks
ssl.keystore.password=changeit
ssl.key.password=changeit
ssl.truststore.location=/etc/kafka/secrets/kafka.truststore.jks
ssl.truststore.password=changeit# SASL mechanisms
sasl.enabled.mechanisms=SCRAM-SHA-512
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
inter.broker.listener.name=SASL_SSL

Docker Compose environment variables

broker:
  image: confluentinc/cp-kafka:7.6.0
  environment:
    KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,SASL_SSL://0.0.0.0:9094
    KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092,SASL_SSL://broker:9094
    KAFKA_SASL_ENABLED_MECHANISMS: SCRAM-SHA-512
    KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: SCRAM-SHA-512
    KAFKA_INTER_BROKER_LISTENER_NAME: SASL_SSL
    KAFKA_LISTENER_NAME_SASL_SSL_SCRAM_SHA_512_SASL_JAAS_CONFIG: >
      org.apache.kafka.common.security.scram.ScramLoginModule required
      username="admin" password="admin-secret";
    KAFKA_SSL_KEYSTORE_LOCATION: /etc/kafka/secrets/broker.keystore.jks
    KAFKA_SSL_KEYSTORE_PASSWORD: changeit

5. Spring Boot — application.yml SASL_SSL config

# application.yml — labs-api with SASL_SSL + SCRAM-SHA-512
spring:
  kafka:
    bootstrap-servers: broker:9094   # SASL_SSL port
    security.protocol: SASL_SSL
    sasl.mechanism: SCRAM-SHA-512
    sasl.jaas.config: >-
      org.apache.kafka.common.security.scram.ScramLoginModule required
      username="labs-api"
      password="${KAFKA_SASL_PASSWORD}";
    ssl:
      trust-store: classpath:kafka.truststore.jks
      trust-store-password: ${KAFKA_TRUSTSTORE_PASSWORD:changeit}
      trust-store-type: JKS

Environment variables for production

# Set in Docker Compose, Kubernetes Secret, or CI/CD
KAFKA_SASL_PASSWORD=labs-api-secret
KAFKA_TRUSTSTORE_PASSWORD=changeit

Verify SASL connection from CLI

# client-sasl-ssl.properties
cat > /tmp/client.properties << EOF
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="labs-api" password="labs-api-secret";
ssl.truststore.location=kafka.truststore.jks
ssl.truststore.password=changeit
EOF

# Test producer
kafka-console-producer.sh \
  --bootstrap-server broker:9094 \
  --topic labs.events \
  --producer.config /tmp/client.properties

# Test consumer
kafka-console-consumer.sh \
  --bootstrap-server broker:9094 \
  --topic labs.events \
  --consumer.config /tmp/client.properties \
  --from-beginning

6. OAuth 2.0 — token-based authentication

For enterprise environments, delegate authentication to an external Identity Provider (Keycloak, Okta, Azure AD).

# server.properties — add OAUTHBEARER mechanism
sasl.enabled.mechanisms=OAUTHBEARER
listener.name.sasl_ssl.oauthbearer.sasl.jaas.config=\
  org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;
listener.name.sasl_ssl.oauthbearer.sasl.server.callback.handler.class=\
  org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerValidatorCallbackHandler
listener.name.sasl_ssl.oauthbearer.sasl.login.callback.handler.class=\
  org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler

# OAuth 2.0 endpoints
sasl.oauthbearer.token.endpoint.url=https://idp.example.com/realms/labs/protocol/openid-connect/token
sasl.oauthbearer.jwks.endpoint.url=https://idp.example.com/realms/labs/protocol/openid-connect/certs
# application.yml — Spring Boot with OAuth 2.0
spring.kafka:
  bootstrap-servers: broker:9094
  security.protocol: SASL_SSL
  sasl.mechanism: OAUTHBEARER
  sasl.jaas.config: >-
    org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required
    oauth.token.endpoint.uri="https://idp.example.com/token"
    oauth.client.id="labs-api"
    oauth.client.secret="${OAUTH_CLIENT_SECRET}";

7. The KRaft SCRAM bootstrap problem — creating the first user before the broker is even reachable

§3’s kafka-configs.sh --alter --add-config 'SCRAM-SHA-512=...' commands connect to the broker itself to create users — which creates a genuine chicken-and-egg problem for a fresh cluster: if SCRAM is the only configured mechanism and no users exist yet, there’s no way to authenticate the very first kafka-configs.sh call that would create the first user.

The fix — pre-provision SCRAM credentials directly into the KRaft metadata log at storage-format time, before the broker ever starts:

# During kafka-storage.sh format (Day 8 §7's cluster initialization step),
# add SCRAM credentials directly — no running broker required yet
kafka-storage.sh format \
  --config /etc/kafka/server.properties \
  --cluster-id $CLUSTER_ID \
  --add-scram 'SCRAM-SHA-512=[name=admin,password=admin-secret]'

Why this matters specifically for KRaft (vs the older ZooKeeper-based setup): in ZooKeeper mode, SCRAM credentials could be written directly to ZooKeeper independent of broker startup order. In KRaft, credentials live in the @metadata topic (Day 8 §2) itself, which doesn’t exist until the cluster is formatted — so --add-scram at format time is the KRaft-native way to solve the bootstrap problem, not an optional convenience. Without it, a fresh cluster configured for SCRAM-only auth with no PLAINTEXT fallback listener has no way to create its first user at all.

8. Authentication without authorization is incomplete

Everything in this material (§2–§7) proves who is connecting — it does not restrict what an authenticated user is allowed to do. Without an authorizer configured, any client with valid credentials (even the labs-socket service account meant only to consume) can produce to any topic, consume from any topic, or perform admin operations.

# Without this, SASL authentication alone provides no access control —
# every authenticated user can do everything
authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer

This is deliberately just a pointer forward, not the full topic. Day 45 covers ACLs in depth — ACL rules, kafka-acls.sh, and per-topic/per-operation permission scoping. The point to internalize now, before moving on: SASL configuration by itself is half of access control, not all of it. A cluster with SCRAM users but no authorizer.class.name configured is authenticated but not actually access-controlled — every valid credential is equally powerful.

9. Credential handling in broker configuration

§4’s server.properties example embeds the admin password directly inline in listener.name.sasl_ssl.scram-sha-512.sasl.jaas.config — readable by anyone who can view the broker’s config file, process list (ps aux can expose JVM args containing inline JAAS strings), or config-management history.

# Prefer a separate JAAS file with restricted OS-level file permissions...
listener.name.sasl_ssl.scram-sha-512.sasl.jaas.config=\
  org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="admin" \
  password="ENV";
  # ...or reference an environment variable / secret manager value, never a literal

Same principle as Day 37 §8’s connector credentials and Day 43 §11’s TLS passwords — this is the SASL-specific instance of a pattern that’s shown up repeatedly across Week 6 and Week 7. Broker config files (and Docker Compose environment blocks) tend to end up in version control, CI logs, or shared debugging output more often than people expect — treat every credential in server.properties/JAAS config the same way, regardless of which specific mechanism it authenticates.

10. Monitoring failed authentication attempts

A sudden spike in failed SASL authentication attempts is a meaningful security signal — either a misconfigured client (worth fixing) or an actual credential-guessing attempt (worth knowing about immediately), and it’s invisible unless you’re specifically watching for it.

# Broker logs record authentication failures — grep for the pattern,
# or better, ship broker logs to a system that can alert on the rate
grep "Authentication failed" /var/log/kafka/server.log

Why this needs its own explicit check, not just general error-log monitoring: authentication failures don’t show up in typical application-level dashboards (consumer lag, producer error rate) — they’re purely broker-side security telemetry, easy to never look at until an actual incident forces someone to go digging through logs after the fact.

11. SCRAM password rotation

Unlike PLAIN’s static JAAS file, SCRAM’s runtime kafka-configs.sh --alter mechanism (§3) makes rotation straightforward — but “straightforward mechanically” still needs a deliberate process, especially for service accounts with many connected client instances.

# Rotate labs-api's password — this OVERWRITES the existing SCRAM credential
kafka-configs.sh --bootstrap-server broker:9092 --alter \
  --add-config 'SCRAM-SHA-512=[iterations=8192,password=new-labs-api-secret]' \
  --entity-type users --entity-name labs-api

The operational gap to plan for: rotating a SCRAM credential takes effect immediately for new connections, but existing long-lived connections using the old password typically keep working until they naturally reconnect (broker restart, network blip, client redeploy) — meaning there’s an ambiguous transition window where some client instances are on the old password and some are on the new one. Unlike TLS certificate rotation (Day 43 §8), SCRAM has no built-in “accept both old and new simultaneously” mechanism — plan credential rotation around a coordinated client redeploy window, not just “run the CLI command and move on.”

12. Common pitfalls

  • Configuring SCRAM-only auth on a fresh KRaft cluster without --add-scram at format time — no way to bootstrap the first user; see §7 for the fix
  • Enabling SASL without an authorizer — authentication alone provides zero access control; every authenticated identity can do everything until authorizer.class.name is set (§8, expanded in Day 45)
  • Inline plaintext passwords in server.properties/JAAS config — same repeated credential-exposure pattern from Day 37/Day 43, here specifically in broker SASL config (§9)
  • Never monitoring failed-authentication-rate — a credential-guessing attempt or a silently misconfigured client retry loop is invisible without this specific check (§10)
  • Rotating a SCRAM password without accounting for the client-transition window — existing long-lived connections don’t automatically pick up the new credential; requires a coordinated redeploy, not just running the CLI command (§11)

SASL mechanism comparison

Key Takeaways

  • TLS encrypts the wire; SASL authenticates the caller — always combine both in production
  • SASL/PLAIN: simple username+password in JAAS config — dev only, no dynamic users
  • SASL/SCRAM-SHA-512 ★: hashed creds in Kafka, add users at runtime via kafka-configs.sh — but a fresh KRaft cluster needs kafka-storage.sh format --add-scram to bootstrap the first user
  • OAuth 2.0: delegate auth to an IdP (Keycloak, Okta) — best for cloud-native stacks
  • Use security.protocol=SASL_SSL (not SASL_PLAINTEXT) — always over TLS
  • SASL authentication alone provides no access control — authorizer.class.name (Day 45) is required to actually restrict what authenticated users can do
  • Never hardcode passwords — use ${ENV_VAR} or Spring Vault in application.yml, and apply the same discipline to broker-side JAAS config
  • Monitor failed-authentication-rate explicitly — it’s invisible in typical application dashboards
  • SCRAM password rotation needs a coordinated client-transition plan — existing connections don’t auto-pick-up new credentials

Support me through GitHub Sponsors.

Thank you for Reading !! See you in the next post.

Next

➡️ Day 45: ACLs — topic-level read/write permissions

Resources

Link to Medium blog

, ,

Related Posts

Kafka

60-Day Kafka 4 Learning Plan · Week 9 — Day 60 of 60

Editorial Team
Kafka

60-Day Kafka 4 Learning Plan · Week 9 — Day 59 of 60

Editorial Team