60-Day Kafka 4 Learning Plan · Week 7 — Day 47 of 60

---

60-Day Kafka 4 Learning Plan · Week 7 — Security & Monitoring Sources: Kafka: The Definitive Guide Ch.10 · github.com/danielqsj/kafka_exporter · prometheus.io/docs

Goal

Deploy kafka_exporter to bridge Kafka JMX metrics into Prometheus format, configure Prometheus to scrape it every 15 seconds, write the key PromQL queries needed for Grafana dashboards in Day 48, and understand the real operational costs — metric cardinality, exporter staleness, storage sizing, and endpoint security — that come with running this monitoring stack in production.

1. Scrape architecture

Kafka broker (JMX :9999)
  → kafka_exporter (:9308/metrics)   ← Prometheus scrapes every 15s
  → Prometheus (:9090)               ← Grafana queries
  → Grafana (:3000)                  → Alerts (Slack / PagerDuty)

kafka_exporter is a Go binary that:

1. Connects to the Kafka broker (not JMX directly — it uses the Kafka client API)
2. Exposes all metrics in Prometheus text format at :9308/metrics
3. Requires no broker changes — runs as a sidecar

2. Docker Compose — kafka_exporter + Prometheus + Grafana

# docker-compose.yml — add monitoring stack
services:
  kafka-exporter:
    image: danielqsj/kafka-exporter:latest
    command:
      - --kafka.server=broker:9092
      - --web.listen-address=:9308
      - --kafka.version=3.0.0
    ports:
      - "9308:9308"
    depends_on:
      - broker
    restart: unless-stopped

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus_data:/prometheus
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.retention.time=15d
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    volumes:
      - grafana_data:/var/lib/grafana
    environment:
      GF_SECURITY_ADMIN_PASSWORD: admin
    depends_on:
      - prometheus
    restart: unless-stopped

volumes:
  prometheus_data:
  grafana_data:

With SASL_SSL (authenticated cluster)

kafka-exporter:
  image: danielqsj/kafka-exporter:latest
  command:
    - --kafka.server=broker:9094
    - --tls.enabled
    - --tls.ca-file=/secrets/ca-cert
    - --tls.cert-file=/secrets/client-cert
    - --tls.key-file=/secrets/client-key
    - --sasl.enabled
    - --sasl.username=kafka-exporter
    - --sasl.password=${KAFKA_EXPORTER_PASSWORD}
    - --sasl.mechanism=SCRAM-SHA-512
  volumes:
    - ./secrets:/secrets:ro

This kafka-exporter principal needs its own ACL grant, per Day 45’s model — DESCRIBE on all topics (prefixed "" for cluster-wide, or scoped to labs. if that’s the only namespace being monitored) and DESCRIBE/READ on consumer groups it reports lag for. Skipping this means the exporter either silently reports incomplete metrics or fails outright once ACLs are enabled, and it’s easy to forget precisely because monitoring tooling doesn’t feel like “a client” the way labs-api does.

3 prometheus.yml — scrape config

# prometheus.yml
global:
  scrape_interval: 15s        # How often to scrape
  evaluation_interval: 15s    # How often to evaluate alerting rules

rule_files:
  - /etc/prometheus/alert_rules.yml   # loaded in Day 48

scrape_configs:
  # Kafka broker metrics via kafka_exporter
  - job_name: kafka
    static_configs:
      - targets: ['kafka-exporter:9308']
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        replacement: kafka-broker

  # Spring Boot labs-api — Micrometer / actuator
  - job_name: labs-api
    metrics_path: /actuator/prometheus
    scrape_interval: 10s
    static_configs:
      - targets: ['labs-api:8080']
    relabel_configs:
      - target_label: application
        replacement: labs-api

  # Spring Boot labs-socket
  - job_name: labs-socket
    metrics_path: /actuator/prometheus
    static_configs:
      - targets: ['labs-socket:8090']

  # Prometheus self-monitoring
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']

Enable Spring Boot Prometheus endpoint

<!-- pom.xml -->
<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

# application.yml
management:
  endpoints:
    web:
      exposure:
        include: health,metrics,prometheus
  metrics:
    tags:
      application: ${spring.application.name}
      environment: production

4. Key Prometheus metric names

Verify scraping is working

# Check kafka_exporter metrics endpoint directly
curl http://localhost:9308/metrics | grep kafka_consumergroup_lag

# Check Prometheus targets (all should be UP)
curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'

5. PromQL queries for Grafana

Consumer lag

# Total lag across all partitions for a group
sum(kafka_consumergroup_lag{consumergroup="labs-api-group"})

# Lag per partition — use in heatmap panel
kafka_consumergroup_lag{topic="labs.events"}

# Lag rate of change (growing = bad)
rate(kafka_consumergroup_lag_sum{consumergroup="labs-api-group"}[5m])

# Max lag across all groups (cluster-wide)
max(kafka_consumergroup_lag_sum)

Throughput

# Messages per second into broker
rate(kafka_server_broker_topic_metrics_messages_in_total[5m])

# Bytes in per second per topic
rate(kafka_server_broker_topic_metrics_bytes_in_total{topic="labs.events"}[5m])

# Producer send rate from Spring Boot actuator
rate(spring_kafka_listener_seconds_count{application="labs-api"}[5m])

Health

# Under-replicated partitions (alert if > 0)
kafka_server_replica_manager_under_replicated_partitions > 0

# Offline partitions (page immediately if > 0)
kafka_controller_kafkacontroller_offline_partitions_count > 0

# Active controller (alert if != 1)
kafka_controller_kafkacontroller_active_controller_count != 1

6. Metric cardinality — the hidden cost of per-partition labels

kafka_consumergroup_lag (§4) carries a label combination of {consumergroup, topic, partition} — for a cluster with many topics, many partitions per topic, and many consumer groups, this multiplies into a genuinely large number of distinct time series, and Prometheus’s storage/query cost scales with cardinality, not just data volume.

Rough cardinality estimate:
  50 topics × 12 partitions avg × 20 consumer groups = 12,000 distinct
  kafka_consumergroup_lag series — before counting any other metric

Each series consumes memory in Prometheus's TSDB continuously, whether
or not anyone ever queries it

Why this matters beyond “more data = more disk”: high-cardinality metrics are the single most common cause of Prometheus performance problems in real deployments — not disk space exhaustion (which is comparatively easy to see coming), but query latency and memory pressure that degrade gradually and are much less obvious to diagnose until dashboards start timing out. If the cluster has a large number of topics/partitions/groups, consider whether every lag series needs to exist at full granularity, or whether some can be aggregated at the exporter/relabeling stage (e.g. dropping labels for internal Streams/ksqlDB-generated groups you don’t dashboard individually) before they ever reach Prometheus’s storage.

7. kafka_exporter’s own limitations — staleness under load

kafka_exporter computes consumer lag by making its own consumer/admin API calls against the cluster on each scrape cycle — it is not a passive JMX bridge, it’s an active client doing real work every 15 seconds (§3’s scrape_interval). Under a cluster with a large number of consumer groups, this polling itself can become a bottleneck, and the lag numbers it reports can lag behind reality more than the dashboard’s timestamp suggests.

Practical implication: don’t treat kafka_consumergroup_lag as a real-time value with sub-scrape-interval precision — it’s an approximation refreshed on the exporter’s own polling cadence, which can itself degrade under load exactly when you’d most want accurate lag data (during an incident). For genuinely time-critical lag monitoring, kafka-consumer-groups.sh --describe (Day 6 §7) queried directly remains the ground truth to cross-check against if a dashboard number looks suspicious.

8. Securing metrics endpoints

Both kafka-exporter:9308/metrics and each Spring Boot app’s /actuator/prometheus are unauthenticated by default in this setup — lower stakes than Day 36/40’s REST APIs (nothing here can mutate state), but the exposed data itself is not nothing: topic names, consumer group names, throughput, and lag numbers reveal real information about internal system architecture and load patterns.

# application.yml — restrict actuator exposure beyond the metrics endpoint itself
management:
  endpoints:
    web:
      exposure:
        include: health,metrics,prometheus   # deliberate allowlist, not "*"
      base-path: /actuator
  endpoint:
    health:
      show-details: when-authorized   # don't leak internal health detail publicly

The right control here is network-level, not endpoint-level auth for most deployments: metrics endpoints are meant to be scraped frequently by an internal system (Prometheus), not accessed interactively — restricting :9308 and /actuator/prometheus to be reachable only from the Prometheus server’s network (firewall rule, Kubernetes NetworkPolicy, or a private subnet) is usually more appropriate than adding auth that Prometheus’s scrape config would then also need to handle. If these endpoints are ever exposed beyond a trusted internal network, add auth — but for the common case, network isolation is the simpler and sufficient control.

9. Prometheus retention & storage sizing

--storage.tsdb.retention.time=15d (§2) is a real disk-sizing decision, not a default to leave unexamined — driven directly by the cardinality discussed in §6.

Rough sizing formula:
  disk usage ≈ (number of active time series) × (bytes per sample) ×
               (samples per day) × (retention days)

A cluster with the 12,000-series estimate from §6, scraped every 15s,
retained 15 days, is a meaningfully different disk footprint than the
same setup with 30-day retention or with cardinality left unmanaged.

Retention is also an incident-investigation tool, not just a cost control — 15 days means you can’t look back further than two weeks to diagnose a slow-building trend (e.g. gradual disk growth on a compacted topic, Day 9 §9’s disk planning concerns showing up months later). Balance retention against actual investigation needs, and consider a separate long-term-storage backend (Prometheus remote-write to a system like Thanos or Mimir) if genuinely long-term trend analysis matters more than what local TSDB retention can practically hold.

10. Recording rules — precomputing expensive, frequently-used queries

sum(kafka_consumergroup_lag_sum) and similar aggregations (§5) get recomputed from raw series every time a dashboard panel or alert rule evaluates them — for a high-cardinality metric (§6), this repeated computation has a real, avoidable cost if the same aggregation is queried often.

# recording_rules.yml — precompute at evaluation_interval, not on every dashboard load
groups:
  - name: kafka_aggregates
    interval: 30s
    rules:
      - record: labs:consumergroup_lag:sum
        expr: sum(kafka_consumergroup_lag_sum) by (consumergroup)

      - record: labs:broker_throughput:rate5m
        expr: rate(kafka_server_broker_topic_metrics_bytes_in_total[5m])

-- Dashboards and alert rules then query the precomputed series directly —
-- cheap, since the expensive aggregation already happened once, on schedule
labs:consumergroup_lag:sum{consumergroup="labs-api-group"}

When this is worth the extra config, not premature optimization: if a query backs multiple dashboard panels, is polled by several alert rules, or aggregates over genuinely high-cardinality raw series, a recording rule turns “recompute this expensive aggregation on every viewer’s page load” into “compute it once every interval, serve the cached result everywhere.” For a small dev cluster with a handful of series, this is unnecessary complexity — it becomes worthwhile as cardinality (§6) and dashboard/alert count grow.

11. Common pitfalls

• Not granting kafka-exporter its own ACLs once ACLs are enabled (Day 45) — the exporter silently reports incomplete metrics or fails to connect, and it’s easy to forget since monitoring tooling doesn’t feel like “a client” (§2)
• Treating kafka_consumergroup_lag as real-time-precise — it’s refreshed on the exporter’s own polling cadence, which can itself degrade under load; cross-check with kafka-consumer-groups.sh --describe when a number looks suspicious during an actual incident (§7)
• Ignoring metric cardinality until Prometheus performance degrades — high cardinality is the most common real-world cause of Prometheus problems, and it creeps up gradually rather than failing loudly (§6)
• Leaving metrics endpoints reachable from outside the trusted network — lower stakes than a mutating REST API, but still real internal-architecture information; network isolation is usually the right control, not necessarily endpoint auth (§8)
• Setting retention without connecting it to actual investigation needs — 15 days might be too short to catch a slow-building trend; decide retention based on how far back you’d realistically need to look, not just a default (§9)

Key Takeaways

• kafka_exporter bridges JMX → Prometheus text format on :9308/metrics
• Prometheus scrapes kafka_exporter every 15s and stores time-series data
• kafka_consumergroup_lag is the most watched metric — alert on sustained growth, but remember it’s an exporter-polled approximation, not a live value
• rate() converts counters to per-second rates for throughput panels
• Metric cardinality (per-topic/partition/group labels) is the real driver of Prometheus storage and query cost — worth actively managing on any cluster with many topics/groups
• The exporter itself needs ACL grants once authorization is enabled, and its own polling can become a bottleneck under high consumer-group counts
• Metrics endpoints should be network-restricted to the Prometheus server, even without full authentication
• Recording rules precompute expensive, frequently-used aggregations — worth adopting once cardinality and dashboard/alert count justify it
• Also scrape Spring Boot /actuator/prometheus for app-level Kafka producer/consumer metrics

---

Support me through GitHub Sponsors.

Thank you for Reading !! See you in the next post.

Next

➡️ Day 48: Grafana — consumer lag dashboards & alerts

Resources

• 📘 Kafka: The Definitive Guide — Chapter 10 (Monitoring)
• 🌐 github.com/danielqsj/kafka_exporter
• 🌐 prometheus.io/docs
• 🌐 Prometheus — recording rules

Link to Medium blog