60-Day Kafka 4 Learning Plan · Week 7 — Day 48 of 60

60-Day Kafka 4 Learning Plan · Week 7 — Security & Monitoring Sources: Kafka: The Definitive Guide Ch.10 · grafana.com/grafana/dashboards/7589 · prometheus.io/docs/alerting

Goal

Build a Grafana dashboard for Kafka health monitoring with six essential panels, configure Prometheus alert rules for under-replicated partitions and consumer lag, wire a Slack notification channel for real-time alerts, and fix the absolute-threshold problem that makes a single hardcoded lag number misleading across topics with different throughput.

1. Add Prometheus datasource

Option A — Grafana UI

Grafana → Connections → Data sources → Add → Prometheus → URL: http://prometheus:9090 → Save & Test

Option B — YAML provisioning (recommended for GitOps)

# provisioning/datasources/prometheus.yml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    access: proxy
    isDefault: true
    editable: false
    jsonData:
      timeInterval: 15s
      httpMethod: POST

Mount into Grafana container:

# docker-compose.yml
grafana:
  image: grafana/grafana:latest
  volumes:
    - ./provisioning:/etc/grafana/provisioning:ro
    - grafana_data:/var/lib/grafana
  environment:
    GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PASSWORD:-admin}
    GF_USERS_ALLOW_SIGN_UP: "false"

Import pre-built dashboard

Grafana → Dashboards → Import → Enter ID 7589 → Load → Select Prometheus datasource → Import

Dashboard 7589 (Kafka Exporter Overview) covers: messages in, bytes in/out, partitions, consumer group lag.

2. Dashboard panels — Kafka overview

Panel 1 — Under-replicated partitions (Stat)

kafka_server_replica_manager_under_replicated_partitions

Panel type: Stat
Thresholds: 0 = green, > 0 = red
Unit: short

Panel 2 — Consumer lag total (Time series)

sum(kafka_consumergroup_lag{consumergroup="labs-api-group"})

Panel type: Time series
Add threshold line at 1000 (warning) and 5000 (critical)
Unit: short (messages)

Panel 3 — Bytes in/sec (Time series)

rate(kafka_server_broker_topic_metrics_bytes_in_total[5m])

Panel type: Time series
Unit: bytes/sec
Split by topic using {topic!="__consumer_offsets"}

Panel 4 — Lag per partition (Heatmap)

kafka_consumergroup_lag{topic="labs.events"}

Panel type: Heatmap
X-axis: time, Y-axis: partition
Instantly spots which partitions are lagging

Panel 5 — Active controller (Stat)

kafka_controller_kafkacontroller_active_controller_count

Panel type: Stat
Thresholds: 1 = green, ≠ 1 = red
Should always read exactly 1

Panel 6 — Messages in/sec (Time series)

rate(kafka_server_broker_topic_metrics_messages_in_total[5m])

Panel type: Time series
Unit: messages/sec
Good for spotting producer traffic drops

3. Alert rules — alert_rules.yml

# alert_rules.yml — load via prometheus.yml rule_files
groups:
  - name: kafka-alerts
    interval: 15s
    rules:

      - alert: KafkaUnderReplicatedPartitions
        expr: kafka_server_replica_manager_under_replicated_partitions > 0
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Kafka under-replicated partitions detected"
          description: "{{ $value }} under-replicated partition(s) on {{ $labels.instance }}"

      - alert: KafkaConsumerLagHigh
        expr: sum(kafka_consumergroup_lag_sum) by (consumergroup) > 1000
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Consumer lag is high for group {{ $labels.consumergroup }}"
          description: "Lag is {{ $value }} messages (threshold: 1000) for 5 minutes"

      - alert: KafkaOfflinePartitions
        expr: kafka_controller_kafkacontroller_offline_partitions_count > 0
        for: 0m   # fire immediately
        labels:
          severity: critical
        annotations:
          summary: "Kafka has offline partitions — messages unreadable"
          description: "{{ $value }} offline partition(s) detected"

      - alert: KafkaActiveControllerMissing
        expr: kafka_controller_kafkacontroller_active_controller_count != 1
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Kafka active controller count is not 1"

Load in prometheus.yml:

rule_files:
  - /etc/prometheus/alert_rules.yml

4. Grafana alert rule (UI) + Slack notification

Create Grafana alert rule

Open consumer lag panel → Edit → Alert → Create alert rule
Query A: sum(kafka_consumergroup_lag_sum{consumergroup="labs-api-group"})
Condition: IS ABOVE 1000
Evaluate every: 1m · For: 5m
Labels: severity=critical, team=platform
Annotations: Summary + description with {{ $value }}

Add Slack contact point

Grafana → Alerting → Contact points → Add contact point → Slack

{
  "type": "slack",
  "settings": {
    "url": "https://hooks.slack.com/services/T.../B.../xxx",
    "title": "🚨 Kafka Alert: {{ .GroupLabels.alertname }}",
    "text": "Consumer lag {{ $value }} > 1000 on {{ $labels.consumergroup }}"
  }
}

Notification policy

Grafana → Alerting → Notification policies → Edit default policy:

Contact point: Slack
Group by: alertname, consumergroup
Group wait: 30s, Repeat interval: 4h

5. Full alerting thresholds

6. The absolute-threshold problem — lag of 1000 doesn’t mean the same thing everywhere

KafkaConsumerLagHigh‘s > 1000 threshold, and Panel 2’s hardcoded consumergroup="labs-api-group" filter, both have the same underlying issue: a single absolute number applied uniformly is either too sensitive for a high-throughput topic or too tolerant for a low-throughput one. This is the same “trend, not absolute value” principle that’s come up repeatedly across this course (Day 19 §9’s migration parity monitoring, Day 28 §10’s dual-write divergence, Day 39 §7’s Redis health) — here applied to the single most common Kafka alert in existence.

labs.events at 5000 msg/sec: a lag of 1000 messages = 0.2 seconds of backlog — utterly trivial
labs.audit-log at 2 msg/sec:  a lag of 1000 messages = 8+ minutes of backlog — genuinely concerning

Better approaches, roughly in order of sophistication:

# 1. Time-to-drain instead of raw message count — normalizes for each topic's own throughput
kafka_consumergroup_lag_sum
  /
clamp_min(rate(kafka_server_broker_topic_metrics_messages_in_total[5m]), 1)
# result is in SECONDS of backlog at current consumption rate — a much more meaningful number

# 2. Per-consumer-group threshold via Prometheus label matching, rather than one global "1000"
# (requires a recording rule or separate alert per group with its own tuned threshold)

# 3. Rate-of-change instead of absolute level — alert when lag is GROWING, not just above a level
deriv(kafka_consumergroup_lag_sum[10m]) > 0

Why the original single-group, single-threshold alert is a trap in practice: it was written to monitor exactly labs-api-group with exactly 1000 — the moment a second consumer group is added (which happens naturally as a system grows, e.g. Day 32’s analytics-service group reading the same topic independently), this alert silently doesn’t cover it at all, and nobody notices until that group’s consumer falls badly behind with zero alerting. Prefer a templated alert across all consumer groups (by (consumergroup) with either a per-group threshold table or the time-to-drain normalization above) over a single hardcoded group name.

7. Testing the alerting pipeline itself

An alert rule that’s never actually fired in testing is an unverified assumption, not a working safety net — a shockingly common real-world gap is discovering during an actual incident that the alert never reaches anyone, because the Slack webhook URL was wrong, or the notification policy’s routing didn’t match the alert’s labels.

# Deliberately trigger a synthetic under-replicated-partitions condition to verify
# the full pipeline: metric → Prometheus rule evaluation → Alertmanager → Slack
# (e.g., stop a broker in a test cluster briefly, or use promtool to test rule logic directly)
promtool test rules alert_rules_test.yml

# alert_rules_test.yml — promtool unit test for the alert rule itself
rule_files:
  - alert_rules.yml
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: 'kafka_server_replica_manager_under_replicated_partitions{instance="broker-1"}'
        values: '0 0 1 1 1 1'   # goes from healthy to under-replicated
    alert_rule_test:
      - eval_time: 5m
        alertname: KafkaUnderReplicatedPartitions
        exp_alerts:
          - exp_labels:
              severity: warning
              instance: broker-1

Why this matters more than it might seem: promtool test rules verifies the Prometheus rule logic is correct without needing a real broker outage — but it doesn’t verify the Slack webhook actually works, or that the notification policy’s group by/routing rules actually deliver this specific alert to the intended channel. Periodically (not just once at setup) trigger a genuine end-to-end test — even something as simple as a deliberately-wrong Slack webhook URL check, or a scheduled “this is a drill” alert — catches configuration drift in the alerting pipeline before a real incident depends on it working.

8. Alert routing & maintenance windows — avoiding alert fatigue

Two related problems worth planning for from the start rather than retrofitting after the team starts ignoring alerts: false pages during planned maintenance, and alert fatigue from thresholds that fire too often to act on.

# Grafana/Alertmanager silence during planned maintenance — mute without disabling the rule
# (via UI: Alerting → Silences → New Silence, matching the relevant labels and a time window)

# Route by severity — warning-level alerts to a low-urgency channel,
# critical alerts to on-call paging, rather than everything to the same Slack channel
route:
  routes:
    - match:
        severity: critical
      receiver: pagerduty-oncall
    - match:
        severity: warning
      receiver: slack-platform-channel

The compounding risk of getting this wrong: a team that receives the same volume of critical-labeled pages for both “the cluster is genuinely down” and “lag briefly spiked during a routine deploy” learns to deprioritize all of them equally — which is exactly the failure mode that turns a well-instrumented system into a false sense of security. Reserve severity: critical + paging for things that genuinely need immediate human intervention (offline partitions, missing controller); route noisier, more marginal signals (a lag threshold that’s still being tuned per §6) to a lower-urgency channel until confidence in the threshold is established.

9. Securing Grafana and Prometheus

Following the exact same pattern established for every other exposed dashboard/API across this course (Connect REST — Day 36 §11, ksqlDB REST — Day 40 §11, Schema Registry — Day 23 §9) — Grafana and Prometheus themselves need access control, not just the Kafka cluster they’re monitoring.

grafana:
  environment:
    GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PASSWORD}   # never leave as default "admin"
    GF_AUTH_ANONYMOUS_ENABLED: "false"                 # disable anonymous dashboard access
    GF_USERS_ALLOW_SIGN_UP: "false"                    # already shown in §1 — keep it

What’s actually at stake here, specifically: a Kafka monitoring dashboard often reveals meaningful information about production traffic patterns, topic names, and consumer group structure — and depending on dashboard configuration, may even allow query-level access to Prometheus that could be used to explore infrastructure topology. Treat Grafana/Prometheus access the same as any other production observability surface: authenticated, and not defaulting to the admin/admin credentials that ship in every quickstart guide.

10. Common pitfalls

A single hardcoded lag threshold and consumer group name — silently stops covering new consumer groups as the system grows, and the same absolute number means very different things for high- vs low-throughput topics (§6)
Never testing that the alerting pipeline actually delivers — a correct Prometheus rule with a broken Slack webhook or misrouted notification policy is functionally the same as no alert at all, and this is only discovered during a real incident unless deliberately tested (§7)
Routing every severity level to the same channel — trains the team to ignore alerts collectively once enough of them turn out to be non-urgent (§8)
Leaving Grafana on default admin credentials with anonymous access enabled — the same exposed-dashboard pattern flagged for every other Week 6/7 REST surface, just for the monitoring layer itself (§9)
Treating for: 0m (fire immediately) as the default for all alerts — appropriate for KafkaOfflinePartitions (genuinely urgent, no value in waiting), but applying zero-delay firing broadly turns every brief, self-resolving blip into a page

Key Takeaways

Provision the Prometheus datasource via YAML for reproducible GitOps setup
Six essential panels: under-replicated, lag total, bytes/sec, lag/partition, controller, msgs/sec
Alert rules in Prometheus (alert_rules.yml) fire before Grafana even renders
for: 5m prevents flapping — condition must hold for full duration before firing
A single absolute lag threshold across all consumer groups is a trap — normalize by throughput (time-to-drain) or template per-group thresholds instead
Test the alerting pipeline end-to-end (promtool test rules plus a real delivery check), not just the Prometheus rule logic in isolation
Route by severity to avoid alert fatigue — critical pages should stay rare and genuinely urgent
Grafana alert rules (UI) add dashboard-linked alerts with richer context
Secure Grafana/Prometheus themselves — same access-control discipline as every other exposed observability/admin surface in this course
Import dashboard ID 7589 from grafana.com for a pre-built Kafka overview

Support me through GitHub Sponsors.

Thank you for Reading !! See you in the next post.

➡️ Day 49: Secure cluster capstone — TLS + SASL + ACLs + monitoring

Resources

📘 Kafka: The Definitive Guide — Chapter 10 (Monitoring)
🌐 grafana.com/grafana/dashboards/7589
🌐 prometheus.io/docs/alerting/latest/alertmanager
🌐 Prometheus — Unit Testing for Rules

Link to Medium blog