60-Day Kafka 4 Learning Plan · Week 7 — Day 46 of 60

---

60-Day Kafka 4 Learning Plan · Week 7 — Security & Monitoring Sources: Kafka: The Definitive Guide Ch.10 · kafka.apache.org/documentation/#monitoring

Goal

Understand how Kafka exposes metrics via JMX, identify the most critical broker and consumer metrics to watch, check consumer lag from the CLI, read MBean paths, and — directly following Week 7’s security material — secure JMX itself rather than undoing three days of TLS/SASL/ACL work with an open monitoring port.

1. What is JMX?

Java Management Extensions (JMX) is the standard Java monitoring interface. Kafka exposes hundreds of metrics as MBeans over a JMX port. Tools like Prometheus kafka_exporter, JConsole, Datadog, and New Relic scrape these metrics.

Three metric families:

2. Enable JMX on the broker

# Shell: set before starting Kafka
export JMX_PORT=9999
bin/kafka-server-start.sh config/server.properties

# Docker Compose
broker:
  image: confluentinc/cp-kafka:7.6.0
  environment:
    KAFKA_JMX_PORT: 9999
    KAFKA_JMX_HOSTNAME: broker   # hostname clients connect to
  ports:
    - "9999:9999"   # expose JMX port

This example as written has no authentication and, per §7/§8, likely doesn’t even work reliably through Docker’s port mapping. Given that Days 43–45 just spent an entire week locking down client access to Kafka (TLS encryption, SASL authentication, ACL authorization), shipping a wide-open, unauthenticated JMX port alongside all of that isn’t a separate, lower-stakes concern — it’s a direct hole in the same security boundary. §7 covers the fix.

Verify JMX is working

# Quick check via jconsole (GUI)
jconsole broker:9999

# Or use kafka-run-class to query a specific MBean
kafka-run-class.sh kafka.tools.JmxTool \
  --jmx-url service:jmx:rmi:///jndi/rmi://broker:9999/jmxrmi \
  --object-name kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions \
  --reporting-interval 1000

3. Key broker MBeans to watch

Full MBean paths

# Replica manager
kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions
kafka.server:type=ReplicaManager,name=IsrShrinkRate

# Controller
kafka.controller:type=KafkaController,name=ActiveControllerCount
kafka.controller:type=KafkaController,name=OfflinePartitionsCount

# Throughput
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec
kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec
kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec

# Request handling
kafka.network:type=SocketServer,name=NetworkProcessorAvgIdlePercent
kafka.network:type=RequestChannel,name=RequestQueueSize
kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce
kafka.network:type=RequestMetrics,name=TotalTimeMs,request=FetchConsumer

Cross-reference — you’ve already seen several of these: UnderReplicatedPartitions and OfflinePartitionsCount were introduced with the ISR mechanics in Day 10 §11; ActiveControllerCount/controller health ties to Day 8 §10’s KRaft monitoring. This day is partly a consolidation of metrics referenced individually across the course, not all-new material — worth recognizing the pattern rather than treating each as freshly introduced.

4. Consumer lag — the most important metric

Consumer lag = latest offset in partition – consumer committed offset

A lag of 0 means the consumer is caught up. Growing lag means the consumer is falling behind producers — a leading indicator of consumer health issues.

Check lag from CLI

# List all consumer groups
kafka-consumer-groups.sh \
  --bootstrap-server broker:9092 \
  --list

# Describe a specific group — shows lag per partition
kafka-consumer-groups.sh \
  --bootstrap-server broker:9092 \
  --describe \
  --group labs-api-group

# Output columns:
# GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID  HOST
# labs-api-group  labs.events     0          1247            1247            0    labs-api-0   /10.0.0.2
# labs-api-group  labs.events     1          890             892             2    labs-api-1   /10.0.0.3
# labs-api-group  labs.events     2          1103            1103            0    labs-api-2   /10.0.0.4

Reset consumer group offset (use with caution)

# Preview reset — what would change
kafka-consumer-groups.sh \
  --bootstrap-server broker:9092 \
  --group labs-api-group \
  --topic labs.events \
  --reset-offsets \
  --to-earliest \
  --dry-run

# Execute reset (group must be inactive)
kafka-consumer-groups.sh \
  --bootstrap-server broker:9092 \
  --group labs-api-group \
  --topic labs.events \
  --reset-offsets \
  --to-earliest \
  --execute

5. JMX MBean paths — read with jconsole or kafka_exporter

# Consumer lag (via JMX)
kafka.consumer:type=consumer-fetch-manager-metrics,
  client-id=labs-api,attribute=records-lag-max

# Producer metrics
kafka.producer:type=producer-metrics,
  client-id=labs-api,attribute=record-send-rate
kafka.producer:type=producer-metrics,
  client-id=labs-api,attribute=record-error-rate
kafka.producer:type=producer-metrics,
  client-id=labs-api,attribute=request-latency-avg

# Request latency on broker
kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce
kafka.network:type=RequestMetrics,name=TotalTimeMs,request=FetchConsumer

Spring Boot actuator metrics (alternative to JMX)

# application.yml — expose Kafka metrics via Spring Boot Actuator
management:
  endpoints:
    web:
      exposure:
        include: health,metrics,prometheus
  metrics:
    tags:
      application: labs-api

# Kafka consumer lag via actuator
curl http://localhost:8080/actuator/metrics/kafka.consumer.records-lag-max

6. Alerting thresholds — what to page on

7. Securing JMX — closing the hole this week’s security work opened a door for

By default, JMX has no authentication and no encryption — anyone who can reach port 9999 gets full read access to every broker metric (including, depending on MBean exposure, some operational internals), and on some JVM/JMX configurations, write access sufficient to trigger administrative actions. Exposing this openly is a real gap sitting right next to the TLS/SASL/ACL hardening from Days 43–45.

# Minimum bar: authentication + SSL for the JMX port itself
export JAVA_OPTS="
  -Dcom.sun.management.jmxremote.authenticate=true
  -Dcom.sun.management.jmxremote.password.file=/etc/kafka/jmxremote.password
  -Dcom.sun.management.jmxremote.access.file=/etc/kafka/jmxremote.access
  -Dcom.sun.management.jmxremote.ssl=true
  -Djavax.net.ssl.keyStore=/etc/kafka/secrets/broker.keystore.jks
  -Djavax.net.ssl.keyStorePassword=changeit
"

# jmxremote.password — restrict OS file permissions to 600, same discipline as any credential file
monitor   readonlypassword
admin     adminpassword

# jmxremote.access — map users to permission levels
monitor   readonly
admin     readwrite

Same minimum bar as every other exposed component across Weeks 6–7 (Connect REST — Day 36 §11, Schema Registry — Day 23 §9, ksqlDB REST — Day 40 §11, Redis — Day 39 §9): if JMX is reachable from anywhere beyond a fully isolated monitoring network, it needs authentication. In practice, many production deployments instead firewall JMX to only the monitoring/Prometheus subnet rather than authenticating every scrape — either approach is acceptable, but “open to the whole network, no auth, no firewall” (as the bare ports: 9999:9999 example implies) is not.

8 The JMX-over-RMI dynamic port problem

ports: 9999:9999 in §2’s Docker Compose looks like it should be sufficient, but JMX over RMI (the default transport) actually needs two ports: the registry port you connect to first (9999), and a second, normally randomly-assigned port that the actual data connection uses after the initial handshake — which Docker’s single port mapping doesn’t expose, and most firewalls block by default since it’s not a fixed, documented port.

# Pin the second RMI port to a known, fixed value so it can actually be
# exposed/firewalled predictably
export JAVA_OPTS="
  -Dcom.sun.management.jmxremote.port=9999
  -Dcom.sun.management.jmxremote.rmi.port=9998
"

broker:
  environment:
    KAFKA_JMX_PORT: 9999
    KAFKA_JMX_HOSTNAME: broker
    JMX_PORT: 9999
  ports:
    - "9999:9999"
    - "9998:9998"   # the RMI data port — must also be exposed/mapped

Why this is a common real-world “JMX works locally but not through Docker/a firewall” complaint: without pinning com.sun.management.jmxremote.rmi.port, the second port changes on every broker restart, making it impossible to reliably firewall or Docker-map — this is precisely why JMX monitoring that works fine when run directly on a host frequently breaks mysteriously once containerized or placed behind network segmentation, and it’s rarely obvious from the error message alone that a second, unpinned port is the actual cause.

9. Per-partition lag vs aggregate — the hot-partition masking risk

§4’s --describe output shows lag per partition, but dashboards and alerts (§6’s “Consumer lag (total) > 1,000”) often aggregate across all partitions in a group — which can hide a real problem.

GROUP           TOPIC        PARTITION  LAG
labs-api-group  labs.events  0          9,800   ← one partition badly behind
labs-api-group  labs.events  1          0
labs-api-group  labs.events  2          0
                                        ─────
                             TOTAL:      9,800   ← looks concerning, but the real
                                                    story is "1 of 3 partitions is
                                                    completely stuck," not "everything
                                                    is slightly behind"

Why this distinction changes the diagnosis: “total lag is high” and “one partition is stuck” point to different root causes — the latter is much more likely to be a hot partition (Day 12 §9/§11) or a stuck consumer thread for that specific partition, not a general throughput problem across the whole consumer group. Alert on max per-partition lag, not just the sum, and always check the per-partition breakdown (§4’s --describe output) before concluding “the consumer needs more capacity” when the real fix might be redistributing a hot key instead.

10. Alerting on trend, not just a static threshold

§6’s thresholds (> 1,000 warning, > 10,000 critical) are a reasonable starting point, but treating them as the only signal misses the same distinction emphasized for migration parity monitoring in Day 19 §9 and Day 28 §10: a stable lag of 1,500 during a known traffic spike is very different from a lag that’s growing from 200 toward 1,000 with no clear cause.

Practical framing, consistent with the pattern used throughout this material: a static threshold catches “something is currently bad”; a trend-aware check (lag increasing over N consecutive samples, or lag failing to drain back toward zero after a traffic spike ends) catches “something is becoming bad” earlier — often with enough lead time to intervene before the static threshold even fires. Where your monitoring tooling supports it (Prometheus alerting rules with a rate()/deriv() function, Day 47), prefer trend-based alerting over a single static cutoff, especially for a metric like lag that has legitimate, expected variance under normal traffic patterns.

11. Common pitfalls

• Exposing JMX with no authentication alongside TLS/SASL/ACL-hardened Kafka access — undermines the exact security posture Days 43–45 just built; treat JMX with the same minimum bar as any other exposed component (§7)
• Assuming a single ports: 9999:9999 mapping is sufficient for JMX-over-RMI — the unpinned second RMI port is a frequent source of “works locally, breaks through Docker/firewall” confusion (§8)
• Alerting only on aggregate consumer lag — masks a hot-partition-specific problem behind a “total looks okay-ish” number; check and alert on max per-partition lag too (§9)
• Treating every threshold breach as equally urgent regardless of trend — a stable, expected lag during a known traffic pattern and a genuinely growing lag look identical to a static threshold check but mean very different things (§10)
• Re-deriving metrics already covered elsewhere without recognizing the overlap — UnderReplicatedPartitions/OfflinePartitionsCount/ActiveControllerCount were introduced in Days 8 and 10; this day consolidates rather than introduces them fresh, worth noticing the pattern when reading monitoring material generally

Key Takeaways

• JMX exposes hundreds of Kafka metrics as MBeans — set JMX_PORT=9999 to enable
• UnderReplicatedPartitions > 0 means data loss risk — alert immediately
• OfflinePartitionsCount > 0 means messages are unreadable — page on-call now
• Consumer lag = latest offset − committed offset; growing lag = consumer falling behind
• JMX has no authentication by default — secure it (auth + SSL, or network isolation) with the same rigor as every other exposed component from Weeks 6–7
• JMX-over-RMI needs a second, pinned port (com.sun.management.jmxremote.rmi.port) to work reliably through Docker or a firewall
• Alert on max per-partition lag, not just the aggregate — a hot partition can hide behind an acceptable-looking total
• Prefer trend-aware alerting over static thresholds where possible — catches problems earlier and reduces noise from expected traffic variance
• Use kafka-consumer-groups.sh --describe for quick lag inspection in dev
• Tomorrow (Day 47): scrape JMX via Prometheus kafka_exporter for dashboards

---

Support me through GitHub Sponsors.

Thank you for Reading !! See you in the next post.

Next

➡️ Day 47: Prometheus — kafka_exporter scrape setup

Resources

• 📘 Kafka: The Definitive Guide — Chapter 10 (Monitoring)
• 🌐 kafka.apache.org/documentation/#monitoring
• 🌐 Oracle — Monitoring and Management Using JMX

Link to Medium blog