60-Day Kafka 4 Learning Plan · Week 8 — Day 56 of 60

---

60-Day Kafka 4 Learning Plan · Week 8 Capstone — Production & Cloud Sources: strimzi.io/docs · keda.sh · Kafka: The Definitive Guide (all chapters)

Goal

Wire everything from Week 8 into a production-grade Kubernetes deployment: Strimzi-managed Kafka 4 KRaft cluster, labs-api producer, labs-socket consumer, HPA on consumer lag — and catch a reintroduced architectural gap this capstone’s own scaling setup creates by not carrying forward Day 39’s WebSocket fan-out fix, then fill in the missing labs-socket-deployment.yaml the deploy commands reference but never show.

Architecture

┌─────────────────────────────────────────────────────────────────-┐
│ namespace: production                                            │
│                                                                  │
│  labs-api (Deployment/HPA)                                       │
│  SCRAM-SHA-512 → SASL_SSL :9093                                  │
│         │ WRITE labs.events                                      │
│         ▼                                                        │
│  ┌─────────────────────────────┐                                 │
│  │  Kafka (Strimzi KRaft)      │   namespace: kafka              │
│  │  3 brokers · 3 controllers  │                                 │
│  │  TLS + SASL + ACLs + JMX    │                                 │
│  └─────────────────────────────┘                                 │
│         │ READ labs.*                                            │
│         ▼                                                        │
│  labs-socket (Deployment/HPA)                                    │
│  SCRAM-SHA-512 → SASL_SSL :9093                                  │
│                                                                  │
│  Monitoring: kafka-exporter → Prometheus → Grafana               │
└───────────────────────────────────────────────────────────────-──┘

1. KafkaTopic + KafkaUser CRDs

# kafka-resources.yaml — topic and users managed by Strimzi entity operator
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: labs-events
  namespace: kafka
  labels:
    strimzi.io/cluster: labs-kafka
spec:
  partitions: 6
  replicas: 3
  config:
    retention.ms: "604800000"     # 7 days
    min.insync.replicas: "2"
    cleanup.policy: delete
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: labs-api
  namespace: kafka
  labels:
    strimzi.io/cluster: labs-kafka
spec:
  authentication:
    type: scram-sha-512
  authorization:
    type: simple
    acls:
      - resource:
          type: topic
          name: labs.events
          patternType: literal
        operations: [Write, Describe, Create]
      - resource:
          type: group
          name: labs-api-group
          patternType: literal
        operations: [Read]
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: labs-socket
  namespace: kafka
  labels:
    strimzi.io/cluster: labs-kafka
spec:
  authentication:
    type: scram-sha-512
  authorization:
    type: simple
    acls:
      - resource:
          type: topic
          name: labs.
          patternType: prefix
        operations: [Read, Describe]
      - resource:
          type: group
          name: labs-socket-group
          patternType: literal
        operations: [Read]

2. labs-api Deployment

# labs-api-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: labs-api
  namespace: production
spec:
  replicas: 2
  selector:
    matchLabels:
      app: labs-api
  template:
    metadata:
      labels:
        app: labs-api
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: /actuator/prometheus
    spec:
      containers:
        - name: labs-api
          image: ghcr.io/anicetkeric/labs-api:latest
          ports:
            - containerPort: 8080
          env:
            - name: KAFKA_BOOTSTRAP_SERVERS
              value: labs-kafka-kafka-bootstrap.kafka.svc:9093
            - name: KAFKA_SECURITY_PROTOCOL
              value: SASL_SSL
            - name: KAFKA_SASL_MECHANISM
              value: SCRAM-SHA-512
            - name: KAFKA_SASL_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: labs-api          # Strimzi-generated secret
                  key: password
            - name: KAFKA_CA_CERT
              valueFrom:
                secretKeyRef:
                  name: labs-kafka-cluster-ca-cert
                  key: ca.crt
          resources:
            requests:
              cpu: 250m
              memory: 512Mi
            limits:
              cpu: 1000m
              memory: 1Gi
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            initialDelaySeconds: 30
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            initialDelaySeconds: 15

3. HPA — scale on consumer lag with KEDA

# keda-scaledobject.yaml — scale labs-socket on consumer lag
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: labs-socket-scaler
  namespace: production
spec:
  scaleTargetRef:
    name: labs-socket
  minReplicaCount: 2
  maxReplicaCount: 6          # max = partition count
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: labs-kafka-kafka-bootstrap.kafka.svc:9092
        consumerGroup: labs-socket-group
        topic: labs.events
        lagThreshold: "100"   # scale up when lag per instance > 100
        offsetResetPolicy: latest

# Install KEDA
helm repo add kedacore https://kedacore.github.io/charts
helm install keda kedacore/keda --namespace keda --create-namespace

minReplicaCount: 2 here silently reintroduces the exact WebSocket multi-instance gap from Day 20 §7 — see §7 for why this matters, and what’s missing to make it safe.

4. Spring Boot application.yml — reads certs from env

# application.yml — labs-api on K8s with Strimzi secrets
spring:
  application.name: labs-api
  kafka:
    bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS}
    security.protocol: ${KAFKA_SECURITY_PROTOCOL:SASL_SSL}
    sasl.mechanism: ${KAFKA_SASL_MECHANISM:SCRAM-SHA-512}
    sasl.jaas.config: >-
      org.apache.kafka.common.security.scram.ScramLoginModule required
      username="labs-api"
      password="${KAFKA_SASL_PASSWORD}";
    ssl:
      trust-store-type: PEM
      trust-store-certificates: ${KAFKA_CA_CERT}

    producer:
      batch-size: 65536
      properties:
        linger.ms: 20
        compression.type: snappy
        acks: all
        enable.idempotence: true

    consumer:
      group-id: labs-api-group
      auto-offset-reset: earliest
      fetch-min-size: 1048576
      properties:
        max.poll.records: 500

management:
  endpoints.web.exposure.include: health,prometheus
  metrics.tags.application: labs-api

5. End-to-end smoke test

#!/bin/bash
# smoke-test.sh — run after every deploy
set -e

API_URL="http://$(kubectl get svc labs-api -n production -o jsonpath='{.status.loadBalancer.ingress[0].ip}')"
KAFKA_POD="labs-kafka-broker-0"
KAFKA_NS="kafka"

echo "=== 1. Produce smoke-test event ==="
RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" \
  -X POST "$API_URL/api/events" \
  -H "Content-Type: application/json" \
  -d '{"type":"smoke-test","payload":"hello-kafka-k8s"}')

[ "$RESPONSE" = "200" ] || [ "$RESPONSE" = "201" ] || {
  echo "FAIL: labs-api returned HTTP $RESPONSE"
  exit 1
}
echo "OK: event produced"

echo "=== 2. Wait for labs-socket to consume ==="
sleep 5
kubectl logs -n production deploy/labs-socket --tail=20 | grep -q "smoke-test" || {
  echo "FAIL: labs-socket did not consume the event"
  exit 1
}
echo "OK: event consumed by labs-socket"

echo "=== 3. Verify consumer lag is 0 ==="
LAG=$(kubectl exec -n $KAFKA_NS $KAFKA_POD -- \
  kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --describe --group labs-api-group 2>/dev/null | \
  awk 'NR>1 {sum+=$6} END {print sum+0}')

[ "$LAG" = "0" ] || {
  echo "WARN: consumer lag = $LAG (not zero)"
}
echo "OK: consumer lag = $LAG"

echo "=== 4. Check under-replicated partitions ==="
URP=$(kubectl exec -n $KAFKA_NS $KAFKA_POD -- \
  kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --under-replicated-partitions 2>/dev/null | wc -l)

[ "$URP" = "0" ] || echo "WARN: $URP under-replicated partitions"

echo ""
echo "=== Smoke test PASSED ==="

This smoke test only proves labs-socket consumed the Kafka message (Step 2’s log grep) — it never verifies the message actually reached a WebSocket client. See §10 for why that gap matters given §7’s finding, and what a more complete check looks like.

6. Week 8 production checklist

Deploy commands

# 1. Deploy Kafka cluster
kubectl apply -f kafka.yaml -n kafka
kubectl apply -f kafkanodepools.yaml -n kafka
kubectl wait kafka/labs-kafka --for=condition=Ready --timeout=300s -n kafka

# 2. Create topics and users
kubectl apply -f kafka-resources.yaml -n kafka

# 3. Deploy application services
kubectl apply -f labs-api-deployment.yaml -n production
kubectl apply -f labs-socket-deployment.yaml -n production

# 4. Deploy KEDA ScaledObjects
kubectl apply -f keda-scaledobject.yaml -n production

# 5. Run smoke test
bash smoke-test.sh

Step 3 applies labs-socket-deployment.yaml, which is never actually shown anywhere in this material — §8 fills it in.

7. The reintroduced WebSocket multi-instance gap — KEDA scaling without a fan-out fix

§3’s KEDA ScaledObject scales labs-socket from minReplicaCount: 2 up to maxReplicaCount: 6 based on consumer lag — meaning this capstone deploys multiple concurrent labs-socket instances as a normal, expected operating condition, not an edge case. This is precisely the scenario Day 20 §7 identified as broken: SimpMessagingTemplate‘s default in-memory broker means a user connected to one labs-socket pod never receives a message consumed by a different pod, and Day 39 built the actual fix (Redis Pub/Sub fan-out) specifically to make multi-instance labs-socket safe.

Nothing in this capstone’s labs-socket-deployment.yaml reference, KEDA config, or architecture diagram mentions Redis at all — meaning as written, this production deployment silently drops WebSocket messages for users connected to a pod other than the one that happened to consume their event, exactly as Day 20 §7 described, the moment KEDA scales beyond 1 replica (which minReplicaCount: 2 guarantees will happen immediately, not just under load).

# What labs-socket-deployment.yaml actually needs — Redis connectivity,
# carried forward from Day 39, not just Kafka SASL/SSL credentials
env:
  - name: SPRING_DATA_REDIS_HOST
    value: labs-redis.production.svc
  - name: SPRING_DATA_REDIS_PORT
    value: "6379"
  - name: SPRING_DATA_REDIS_PASSWORD
    valueFrom:
      secretKeyRef:
        name: labs-redis-credentials
        key: password

Why this is worth catching specifically at the capstone stage, not just noting “Day 39 covered this already”: a capstone is where isolated day-by-day lessons get assembled into one deployment, and exactly this kind of integration gap — where a fix from one earlier day quietly gets dropped when a later day’s material is assembled independently — is the realistic failure mode for a real team’s actual production rollout too. The fact that labs-socket here has SASL/SSL Kafka credentials (correctly carried forward from Days 43-45) but no Redis credentials (silently dropped from Day 39) is a good illustration of why an integration review pass across all assembled pieces matters, not just verifying each piece individually looks correct.

8. Filling in the missing labs-socket-deployment.yaml

The deploy commands (end of this document) reference labs-socket-deployment.yaml directly, but it’s never shown — here’s the corrected version, mirroring labs-api‘s pattern (§2) and including the Redis connectivity from §7:

# labs-socket-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: labs-socket
  namespace: production
spec:
  replicas: 2
  selector:
    matchLabels:
      app: labs-socket
  template:
    metadata:
      labels:
        app: labs-socket
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8090"
        prometheus.io/path: /actuator/prometheus
    spec:
      containers:
        - name: labs-socket
          image: ghcr.io/anicetkeric/labs-socket:latest
          ports:
            - containerPort: 8090
          env:
            - name: KAFKA_BOOTSTRAP_SERVERS
              value: labs-kafka-kafka-bootstrap.kafka.svc:9093
            - name: KAFKA_SECURITY_PROTOCOL
              value: SASL_SSL
            - name: KAFKA_SASL_MECHANISM
              value: SCRAM-SHA-512
            - name: KAFKA_SASL_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: labs-socket        # Strimzi-generated secret for THIS user
                  key: password
            - name: KAFKA_CA_CERT
              valueFrom:
                secretKeyRef:
                  name: labs-kafka-cluster-ca-cert
                  key: ca.crt
            # Redis — required for safe multi-instance WebSocket fan-out (Day 39, §7 above)
            - name: SPRING_DATA_REDIS_HOST
              value: labs-redis.production.svc
            - name: SPRING_DATA_REDIS_PORT
              value: "6379"
            - name: SPRING_DATA_REDIS_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: labs-redis-credentials
                  key: password
          resources:
            requests:
              cpu: 250m
              memory: 512Mi
            limits:
              cpu: 1000m
              memory: 1Gi
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8090
            initialDelaySeconds: 30
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8090
            initialDelaySeconds: 15

Note the credential is labs-socket, not labs-api — each Strimzi KafkaUser from §1 generates its own uniquely-named secret; mounting the wrong one (an easy copy-paste mistake when building this file by adapting §2’s) would authenticate labs-socket as the wrong principal, with ACL permissions (§1) that don’t match what it actually needs to do.

9. KEDA lag-based autoscaling — the cold-start dynamic worth knowing

§3’s lagThreshold: "100" scales up when per-instance lag exceeds 100 — a reasonable-looking trigger, but scaling itself has a side effect worth understanding: adding a new labs-socket replica means a new member joining the labs-socket-group consumer group, which triggers a rebalance (Day 6 §4) — and a rebalance transiently pauses consumption across the whole group while partition reassignment completes.

The practical consequence: a scale-up event triggered by rising lag can, for a brief window, make lag look worse immediately after scaling (rebalance pause) before it gets better (more consumer capacity now online) — this is normal and expected, not a sign the scaling decision was wrong, but it’s worth knowing when interpreting a lag graph around a scaling event, and worth factoring into how aggressively KEDA’s cooldown/polling interval is tuned (too aggressive a scale-up policy can trigger repeated rebalances before each one has a chance to stabilize, a self-inflicted thrashing pattern). Static membership (Day 6 §4’s group.instance.id) doesn’t help here the way it helps with brief disconnects — a KEDA scale-up is a genuine membership change, not a reconnect, so a rebalance is unavoidable; the goal is just not triggering it more often than necessary.

10. A more complete smoke test — verifying actual WebSocket delivery

§5’s Step 2 (grep -q "smoke-test" against labs-socket logs) only proves the Kafka consumer received and logged the event — exactly the same category of gap flagged for the naive tests in Day 21 §5 and Day 28 §8, just at the capstone/production level instead of a unit test. Combined with §7’s finding, this smoke test would actually pass even in a broken deployment where WebSocket clients never receive real-time updates, as long as labs-socket‘s own log line fires.

echo "=== 2b. Verify actual WebSocket delivery, not just consumption ==="
# Requires a lightweight STOMP test client (same pattern as Day 20 §11's test)
WS_RESULT=$(node ws-smoke-client.js --url "ws://$SOCKET_URL/ws" --topic /topic/events --timeout 10)
[ "$WS_RESULT" = "received" ] || {
  echo "FAIL: WebSocket client did not receive the smoke-test event"
  exit 1
}
echo "OK: WebSocket delivery confirmed end-to-end"

Why this specific addition matters more here than in an ordinary unit test context: this smoke test runs against the real, scaled, multi-instance production deployment — it’s exactly positioned to catch the §7 gap in practice (the WebSocket client connects to whichever pod the load balancer routes it to, which may not be the pod that consumed the message) if it actually tested the full path. Running only the Kafka-consumption check gives false confidence precisely because it passes both when the deployment is healthy and when it’s suffering from exactly the bug this capstone silently reintroduced.

11. Common pitfalls

• Deploying labs-socket with minReplicaCount: 2+ without carrying forward Day 39’s Redis Pub/Sub fan-out — silently drops WebSocket messages for users on a different pod than the one that consumed their event, the single most consequential gap in this capstone (§7)
• Referencing a deployment file (labs-socket-deployment.yaml) in deploy commands without ever defining it — a documentation gap that, if copied as-is, leaves the reader to guess or improvise the missing file under time pressure (§8)
• Mounting the wrong Strimzi-generated secret (e.g. labs-api‘s instead of labs-socket‘s) when adapting one Deployment YAML into another — authenticates as the wrong principal with the wrong ACL grants (§8)
• Interpreting a lag spike immediately after a KEDA scale-up as a scaling failure — a brief rebalance-induced pause is normal and expected, not evidence the trigger was miscalibrated (§9)
• Trusting a smoke test that only checks Kafka-side consumption — passes even when the actual user-facing delivery path (WebSocket) is broken, giving false confidence in exactly the scenario §7 describes (§10)

Key Takeaways

• KafkaTopic and KafkaUser CRDs keep topic/ACL config in Git — GitOps for Kafka
• Mount Strimzi-generated secrets directly into Deployment env vars — no hardcoded passwords, but double-check each Deployment mounts its own KafkaUser‘s secret, not a copy-pasted neighbor’s
• KEDA ScaledObject on consumer lag lets labs-socket auto-scale when events spike — but scaling labs-socket beyond one replica is exactly the condition Day 20 §7 and Day 39 exist to make safe, and this capstone’s labs-socket-deployment.yaml needs Redis connectivity carried forward to actually be safe at minReplicaCount: 2
• A KEDA scale-up triggers a real consumer group rebalance — expect a brief lag blip immediately after scaling, not a sign of misconfiguration
• Always run a smoke test post-deploy that verifies the full user-facing path (WebSocket delivery), not just Kafka-side consumption — the latter can pass even when the former is silently broken
• Week 8 covers the full production arc: deploy → size → tune → cloud → replicate → recover — and a capstone is exactly where gaps between independently-correct pieces get caught, if reviewed as an assembled whole rather than piece by piece

---

Support me through GitHub Sponsors.

Thank you for Reading !! See you in the next story.

Next

➡️ Week 9, Day 57: Ecosystem review — revisit all 8 weeks, fill gaps

Resources

• 🌐 strimzi.io/docs
• 🌐 keda.sh
• 📘 Kafka: The Definitive Guide — all chapters