60-Day Kafka 4 Learning Plan · Week 7 — Day 45 of 60

---

60-Day Kafka 4 Learning Plan · Week 7 — Security & Monitoring Sources: Kafka: The Definitive Guide Ch.12 · kafka.apache.org/documentation/#security_authz

Goal

Lock down Kafka so that only authorised principals can perform specific operations on specific topics. Grant labs-api WRITE access to labs.events, grant labs-socket READ access to all labs.* topics, deny everything else by default, understand DENY-vs-ALLOW evaluation order, extend the ACL model to Kafka Streams/ksqlDB/Connect’s internal topics, and monitor authorization denials.

1. What are ACLs?

An ACL (Access Control List) entry has four components:

Default behaviour: when the authorizer is enabled, all access is denied unless an explicit ALLOW ACL exists. This is the principle of least privilege.

TLS + SASL identify the principal. ACLs then decide what that principal may do.

2. Enable ACLs on the broker

# server.properties — Kafka 4 KRaft
# Activate the StandardAuthorizer (KRaft-native, replaces ZooKeeper AclAuthorizer)
authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer

# Super users bypass all ACL checks — keep minimal
super.users=User:admin;User:ANONYMOUS

# Allow brokers to communicate before ACLs are fully loaded
allow.everyone.if.no.acl.found=false

Docker Compose environment variables

broker:
  environment:
    KAFKA_AUTHORIZER_CLASS_NAME: org.apache.kafka.metadata.authorizer.StandardAuthorizer
    KAFKA_SUPER_USERS: "User:admin;User:ANONYMOUS"
    KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "false"

Admin properties file (used by kafka-acls.sh)

# /tmp/admin.properties
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="admin" password="admin-secret";
ssl.truststore.location=kafka.truststore.jks
ssl.truststore.password=changeit

3. Grant labs-api WRITE access to labs.events

Producers need WRITE + DESCRIBE on the topic, and READ on their consumer group (needed internally for idempotent producers).

# WRITE + DESCRIBE on labs.events
kafka-acls.sh \
  --bootstrap-server broker:9094 \
  --command-config /tmp/admin.properties \
  --add \
  --allow-principal User:labs-api \
  --operation Write \
  --operation Describe \
  --topic labs.events

# READ on own consumer group (required for idempotent producers)
kafka-acls.sh \
  --bootstrap-server broker:9094 \
  --command-config /tmp/admin.properties \
  --add \
  --allow-principal User:labs-api \
  --operation Read \
  --group labs-api-group

# Also allow CREATE topic if labs-api auto-creates topics
kafka-acls.sh \
  --bootstrap-server broker:9094 \
  --command-config /tmp/admin.properties \
  --add \
  --allow-principal User:labs-api \
  --operation Create \
  --topic labs.events

4. Grant labs-socket READ on all labs.* topics

Use --resource-pattern-type prefixed to cover all current and future labs.* topics with a single ACL entry.

# READ + DESCRIBE on all topics matching prefix "labs."
kafka-acls.sh \
  --bootstrap-server broker:9094 \
  --command-config /tmp/admin.properties \
  --add \
  --allow-principal User:labs-socket \
  --operation Read \
  --operation Describe \
  --topic labs. \
  --resource-pattern-type prefixed

# READ on consumer group
kafka-acls.sh \
  --bootstrap-server broker:9094 \
  --command-config /tmp/admin.properties \
  --add \
  --allow-principal User:labs-socket \
  --operation Read \
  --group labs-socket-group

5. List, verify and revoke ACLs

# List ALL ACLs in the cluster
kafka-acls.sh \
  --bootstrap-server broker:9094 \
  --command-config /tmp/admin.properties \
  --list

# List ACLs for a specific principal
kafka-acls.sh \
  --bootstrap-server broker:9094 \
  --command-config /tmp/admin.properties \
  --list \
  --principal User:labs-api

# List ACLs on a specific topic
kafka-acls.sh \
  --bootstrap-server broker:9094 \
  --command-config /tmp/admin.properties \
  --list \
  --topic labs.events

# REVOKE WRITE on labs.events from labs-api
kafka-acls.sh \
  --bootstrap-server broker:9094 \
  --command-config /tmp/admin.properties \
  --remove \
  --allow-principal User:labs-api \
  --operation Write \
  --topic labs.events

# DENY a principal explicitly (stronger than missing ALLOW)
kafka-acls.sh \
  --bootstrap-server broker:9094 \
  --command-config /tmp/admin.properties \
  --add \
  --deny-principal User:rogue-service \
  --operation All \
  --topic labs.events

6. labs-api ACL matrix

7. Operations reference

8. DENY always wins — ACL evaluation order semantics

The §5 example adds a --deny-principal ACL “on top of” whatever ALLOW rules might exist — worth being explicit about exactly how that interacts, since it’s a common source of “why is this still blocked” confusion.

Evaluation order (not the order ACLs were created in):
  1. Is there ANY matching DENY ACL for this principal/resource/operation? → if yes, DENY, full stop.
  2. Is there a matching ALLOW ACL? → if yes, ALLOW.
  3. No matching ACL at all? → DENY (the default-deny principle from §1).

DENY is not “the opposite of the most recent ALLOW” — it’s evaluated first, unconditionally, regardless of when either rule was added or which is “more specific.” A principal with a broad ALLOW ... Read ... Topic:labs. (prefixed) and a narrower DENY ... Read ... Topic:labs.sensitive-topic will be denied on labs.sensitive-topic specifically, and this works correctly — but if someone later adds a second narrower ALLOW trying to re-permit just that one topic, it still loses to the existing DENY. To reverse a DENY, you must explicitly remove it (--remove --deny-principal ...), not attempt to out-specify it with another ALLOW.

9. ACLs for Kafka Streams, ksqlDB, and Connect — the gap this matrix doesn’t cover

§6’s matrix works for simple producer/consumer services like labs-api/labs-socket, but a Kafka Streams app (Week 5), ksqlDB (Day 40 §8), or Connect (Week 6) needs a meaningfully broader ACL footprint — because each of those creates and manages internal topics on the fly (repartition topics, changelog topics, _connect-configs/_connect-offsets/_connect-status) that a simple “READ this topic, WRITE that topic” grant doesn’t cover.

# Kafka Streams / ksqlDB application principal needs, at minimum:

# READ + WRITE + DESCRIBE + CREATE + DELETE on its own internal topics,
# scoped by the application-id / KSQL_KSQL_SERVICE_ID prefix (Day 29 §9, Day 40 §8)
kafka-acls.sh --bootstrap-server broker:9094 --command-config /tmp/admin.properties \
  --add --allow-principal User:labs-streams-app \
  --operation Read --operation Write --operation Describe --operation Create --operation Delete \
  --topic labs-streams-app- \
  --resource-pattern-type prefixed

# READ + WRITE on its own consumer group (it IS a consumer group — Day 29 §10)
kafka-acls.sh --bootstrap-server broker:9094 --command-config /tmp/admin.properties \
  --add --allow-principal User:labs-streams-app \
  --operation Read \
  --group labs-streams-app

# If using transactions/EOS (Day 11, Day 29 exactly_once_v2): the transactional.id itself
# also needs an explicit grant — this is easy to miss since it's a different resource TYPE
kafka-acls.sh --bootstrap-server broker:9094 --command-config /tmp/admin.properties \
  --add --allow-principal User:labs-streams-app \
  --operation Write --operation Describe \
  --transactional-id labs-streams-app \
  --resource-pattern-type prefixed

Why this is worth its own section, not just “same as before but bigger”: a team that correctly ACL’d their simple producer/consumer services and then deployed a Kafka Streams or ksqlDB app with the same narrow pattern will see it fail at startup with authorization errors on topics they never explicitly created — the repartition/changelog topics (Day 29 §8, §9) that the app creates for itself still need CREATE/WRITE/READ permission, because from the authorizer’s perspective, the app creating its own internal topic is just another client action requiring authorization like any other. Scope the ACL to the application-id/service-id prefix rather than trying to enumerate every internal topic name individually — they’re not stable across topology changes (Day 29 §9’s warning about auto-generated names applies here too).

10. Testing ACLs — verifying actual denial, not just successful grants

It’s easy to test that an ALLOW works (the client connects and it just works) and never verify that a DENY actually denies — worth deliberately testing the negative case, not just the happy path.

# Attempt to produce as labs-socket (which only has READ, not WRITE) — should fail
cat > /tmp/labs-socket-client.properties << EOF
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="labs-socket" password="labs-socket-secret";
ssl.truststore.location=kafka.truststore.jks
ssl.truststore.password=changeit
EOF

kafka-console-producer.sh \
  --bootstrap-server broker:9094 \
  --topic labs.events \
  --producer.config /tmp/labs-socket-client.properties
# Expect: org.apache.kafka.common.errors.TopicAuthorizationException

Make this an actual automated check, not a one-time manual verification. A CI or staging-environment test that attempts a known-forbidden operation and asserts it fails with TopicAuthorizationException/GroupAuthorizationException catches ACL regressions (an overly broad grant accidentally added, or a DENY accidentally removed) the same way any other security-relevant behavior deserves regression testing — an ACL policy that’s only ever tested via “does the legitimate path work” never catches “does the illegitimate path correctly NOT work.”

11. Monitoring authorization denials

Building directly on Day 44 §10’s failed-authentication monitoring — authorization denials (a valid, authenticated principal attempting something it’s not permitted to do) are a distinct, equally important signal.

# Broker logs authorization denials distinctly from authentication failures
grep "Principal.*not authorized" /var/log/kafka/server.log

The combined picture with Day 44 §10: failed authentication tells you someone couldn’t prove who they are; authorization denial tells you someone did prove who they are and then tried something they’re not allowed to do. Both deserve monitoring, but a spike in the second is generally more concerning — it means a legitimate, valid credential is being used to attempt something unauthorized, which is a step further into “this might be an actual problem” than a failed login attempt.

12. Common pitfalls

• Forgetting DESCRIBE — a principal with WRITE but no DESCRIBE on a topic gets a confusing UNKNOWN_TOPIC_OR_PARTITION-style error instead of a clear authorization error, since metadata lookup fails before the actual write is even attempted; always grant DESCRIBE alongside READ/WRITE
• Trying to “out-specify” a DENY with a narrower ALLOW — DENY always wins regardless of specificity or add order; the only way to reverse it is to remove the DENY explicitly (§8)
• Applying the simple producer/consumer ACL pattern to a Kafka Streams/ksqlDB/Connect app — these need CREATE/WRITE/READ/DELETE on their own internal topics (prefixed by application-id) and, for EOS, an explicit --transactional-id grant — none of which the basic pattern covers (§9)
• Only testing that ALLOW grants work, never that DENY/missing-ACL actually blocks — an ACL policy is a security control; test its negative case with the same rigor as its positive case (§10)
• Monitoring authentication failures (Day 44 §10) but not authorization denials — they’re distinct signals requiring distinct log greps/alerts, and a spike in the latter is generally the more urgent one (§11)

Key Takeaways

• ACLs define WHO can do WHAT on WHICH resource — default is deny all
• Enable with authorizer.class.name=StandardAuthorizer in server.properties
• super.users bypass all ACL checks — keep this list minimal (just admin)
• Producers need WRITE + DESCRIBE on the topic AND READ on their consumer group
• Use --resource-pattern-type prefixed for wildcard coverage (labs. covers all labs.*)
• DENY always wins over ALLOW, regardless of rule order or specificity — reverse a DENY by removing it, not by adding a competing ALLOW
• Kafka Streams/ksqlDB/Connect need broader ACLs than simple producers/consumers — internal topics (repartition, changelog, _connect-*) and transactional IDs need their own explicit grants
• Test the negative case (a known-forbidden operation correctly fails) as deliberately as the positive case
• Monitor both failed authentication (Day 44 §10) and authorization denials — distinct signals, both worth alerting on
• ACLs are stored in Kafka metadata — no broker restart needed after changes

---

Support me through GitHub Sponsors.

Thank you for Reading !! See you in the next post.

Next

➡️ Day 46: JMX metrics — key broker & consumer-lag metrics

Resources

• 📘 Kafka: The Definitive Guide — Chapter 12 (Security)
• 🌐 kafka.apache.org/documentation/#security_authz
• 🌐 Kafka Streams — Required ACLs

Link to Medium blog