60-Day Kafka 4 Learning Plan · Week 8— Day 50 of 60

60-Day Kafka 4 Learning Plan · Week 8 — Production & Cloud Sources: Kafka: The Definitive Guide Ch.2 · strimzi.io/docs · strimzi.io/blog/2023/10/kraft-mode

Goal

Deploy Kafka 4 in KRaft mode on Kubernetes using the Strimzi operator. Understand KafkaNodePool, StatefulSets, persistent volumes, and PodDisruptionBudgets — the building blocks of a production-grade K8s Kafka cluster — and fix a version/config mismatch in the original example that doesn’t actually match Kafka 4 KRaft-only semantics.

1. Why Strimzi?

Strimzi is a CNCF-graduated operator that manages the full Kafka lifecycle on Kubernetes. Instead of manually wiring StatefulSets, Services, ConfigMaps, and Secrets, you declare a Kafka custom resource and the operator does the rest.

2. Install Strimzi operator

Option A — Helm (recommended)

# Add Strimzi Helm chart repo
helm repo add strimzi https://strimzi.io/charts/
helm repo update

# Install operator in kafka namespace
helm install strimzi-operator strimzi/strimzi-kafka-operator \
  --namespace kafka \
  --create-namespace \
  --version 0.40.0 \
  --set watchNamespaces="{kafka}"

# Verify operator is running
kubectl get pods -n kafka
# NAME                                        READY   STATUS    RESTARTS
# strimzi-cluster-operator-xxx-yyy            1/1     Running   0

Version note: Strimzi 0.40.0 (used here for the operator itself) predates broad Kafka 4.0 support — Strimzi’s own release notes specify which Kafka versions each operator version supports. Before deploying, check Strimzi’s supported-versions table and pick an operator release that explicitly lists Kafka 4.0.x, rather than assuming any recent-looking operator version works with any recent-looking Kafka version. §8 covers a related version mismatch in the Kafka CR itself.

Option B — kubectl apply

kubectl create namespace kafka
kubectl apply -f "https://strimzi.io/install/latest?namespace=kafka" -n kafka

3. Kafka CR — KRaft mode

# kafka.yaml — KRaft cluster
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: labs-kafka
  namespace: kafka
  annotations:
    strimzi.io/node-pools: enabled   # required for KafkaNodePool
    strimzi.io/kraft: enabled        # enable KRaft mode
spec:
  kafka:
    version: 3.7.0
    metadataVersion: 3.7-IV4
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: tls
      - name: external
        port: 9094
        type: loadbalancer
        tls: true
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.7"
    resources:
      requests:
        memory: 4Gi
        cpu: "1"
      limits:
        memory: 8Gi
        cpu: "2"
    jvmOptions:
      -Xms: 2048m
      -Xmx: 4096m
  entityOperator:
    topicOperator: {}   # enables KafkaTopic CRD
    userOperator: {}    # enables KafkaUser CRD

This example is pinned to Kafka 3.7, not Kafka 4 — see §8 for the corrected version. Two things need to change for an actual Kafka 4 deployment: version/metadataVersion need to target a 4.0.x release, and inter.broker.protocol.version is a ZooKeeper-era concept that Kafka 4’s KRaft-only architecture doesn’t use the same way — §8 explains what replaces it.

Apply:

kubectl apply -f kafka.yaml -n kafka
kubectl wait kafka/labs-kafka --for=condition=Ready --timeout=300s -n kafka

4. KafkaNodePool — broker + controller pools

# kafkanodepool-broker.yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: broker
  namespace: kafka
  labels:
    strimzi.io/cluster: labs-kafka
spec:
  replicas: 3
  roles:
    - broker
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        deleteClaim: false        # keeps PVC on pod delete — critical for prod!
        class: standard-rwo       # use SSD-backed StorageClass
  resources:
    requests:
      memory: 4Gi
      cpu: "1"
    limits:
      memory: 8Gi
      cpu: "2"
---
# kafkanodepool-controller.yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: controller
  namespace: kafka
  labels:
    strimzi.io/cluster: labs-kafka
spec:
  replicas: 3
  roles:
    - controller
  storage:
    type: persistent-claim
    size: 10Gi          # controllers need much less storage than brokers
    deleteClaim: false

5. K8s resources Strimzi creates

6. KafkaTopic and KafkaUser CRDs

# topic.yaml — managed topic
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: labs-events
  namespace: kafka
  labels:
    strimzi.io/cluster: labs-kafka
spec:
  partitions: 6
  replicas: 3
  config:
    retention.ms: 604800000    # 7 days
    cleanup.policy: delete

# user.yaml — managed user with SCRAM auth + ACLs
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: labs-api
  namespace: kafka
  labels:
    strimzi.io/cluster: labs-kafka
spec:
  authentication:
    type: scram-sha-512
  authorization:
    type: simple
    acls:
      - resource:
          type: topic
          name: labs.events
          patternType: literal
        operations: [Write, Describe]
      - resource:
          type: group
          name: labs-api-group
          patternType: literal
        operations: [Read]

The operator creates a K8s Secret labs-api with the generated password — mount it into your Spring Boot pod.

Familiar pattern: this KafkaUser CRD is declaratively expressing exactly the SCRAM credential creation (Day 44 §3) and ACL grants (Day 45 §3, §6) covered manually via kafka-configs.sh/kafka-acls.sh — Strimzi’s operator runs those same underlying operations for you and reconciles them continuously against the CR, rather than being a different security model.

7. Connect Spring Boot to Strimzi cluster

# application.yml — read SASL password from K8s Secret
spring:
  kafka:
    bootstrap-servers: labs-kafka-kafka-bootstrap.kafka.svc:9093
    security.protocol: SASL_SSL
    sasl.mechanism: SCRAM-SHA-512
    sasl.jaas.config: >-
      org.apache.kafka.common.security.scram.ScramLoginModule required
      username="labs-api"
      password="${KAFKA_PASSWORD}";
    ssl:
      trust-store-type: PEM
      trust-store-certificates: ${KAFKA_CA_CERT}

# deployment.yaml excerpt — mount Strimzi-generated secret
env:
  - name: KAFKA_PASSWORD
    valueFrom:
      secretKeyRef:
        name: labs-api      # KafkaUser secret
        key: password
  - name: KAFKA_CA_CERT
    valueFrom:
      secretKeyRef:
        name: labs-kafka-cluster-ca-cert
        key: ca.crt

8. Correcting the version mismatch — Kafka 4 and metadata.version

The Kafka CR in §3 targets Kafka 3.7, and sets inter.broker.protocol.version: "3.7" — neither of which is right for an actual Kafka 4 deployment on Strimzi.

spec:
  kafka:
    version: 4.0.0
    metadataVersion: 4.0-IV3   # check Strimzi's release notes for the exact IV level shipped with your operator version
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      # inter.broker.protocol.version REMOVED — see explanation below

Why inter.broker.protocol.version doesn’t belong in a Kafka 4 KRaft-only config at all: this setting existed to let a mixed-version cluster (some brokers on an older Kafka release, some newer) negotiate a common wire protocol during a rolling upgrade — a ZooKeeper-era concern from when broker software versions and metadata format were only loosely coupled. In KRaft, that role is played entirely by metadata.version (what Strimzi’s metadataVersion field sets) — it’s the single source of truth for the cluster’s active feature/protocol level, coordinated through the @metadata log (Day 8 §2) rather than a separate per-broker config. Setting inter.broker.protocol.version in a Kafka 4 KRaft cluster is at best a no-op and at worst a configuration Strimzi/Kafka will reject outright, depending on version — remove it rather than trying to keep it “just in case.”

9. Rack awareness — the table promised it, here’s how to actually configure it

§1’s feature table lists “rack awareness” as something Strimzi handles, but nothing in §3/§4 actually configures it — worth closing that gap, since it’s the direct Kubernetes-native expression of Day 10 §8’s broker.rack concept.

# kafkanodepool-broker.yaml — add rack awareness
spec:
  # ... replicas, roles, storage, resources as before
  template:
    pod:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  strimzi.io/cluster: labs-kafka
                  strimzi.io/kind: Kafka
              topologyKey: topology.kubernetes.io/zone   # spread brokers across AZs
---
# In the Kafka CR itself
spec:
  kafka:
    rack:
      topologyKey: topology.kubernetes.io/zone   # sets broker.rack from the node's actual AZ label

Why this matters exactly as much on Kubernetes as it did bare-metal (Day 10 §8): without this, Kubernetes’ scheduler has no Kafka-specific reason to avoid placing all 3 broker replicas in the same availability zone — replicas: 3 alone guarantees pod count, not failure-domain spread. The rack.topologyKey setting is what makes Strimzi actually populate broker.rack per pod based on its real node’s zone label, restoring the same AZ-aware replica placement guarantee from Day 10 §8, just derived from Kubernetes topology labels instead of manually set per broker.

10. PodDisruptionBudget and JVM heap vs container memory

§5’s table mentions a PodDisruptionBudget ensuring “min 2 brokers stay up during upgrades,” and §3/§4 set both resources (container memory) and jvmOptions (JVM heap) — two settings that need to agree with each other, not be picked independently.

# The relationship that matters: JVM heap should leave real headroom below the
# container memory limit — RocksDB (Day 29 §8), page cache, and JVM overhead
# outside the heap all need space too
resources:
  requests:
    memory: 4Gi
  limits:
    memory: 8Gi          # container memory ceiling
jvmOptions:
  -Xms: 2048m
  -Xmx: 4096m             # heap ceiling — well under the 8Gi container limit, correctly

The mistake this example avoids, worth calling out explicitly: setting -Xmx close to or equal to the container memory limit starves the JVM of room for off-heap memory (Kafka’s network buffers, page cache benefiting from OS-level file caching, RocksDB’s own memory needs from Day 34 §8 if this pod also runs Streams workloads) — a container that hits its memory limit gets OOM-killed by Kubernetes regardless of what the JVM itself thinks its budget is. The 4Gi limit / 4Gi max-heap gap shown here (heap capped at half the container limit) is a reasonable starting ratio, not an arbitrary choice — validate it against actual observed memory usage rather than assuming any single ratio is universally correct.

For the PodDisruptionBudget itself, Strimzi manages it automatically based on replicas — no separate YAML needed, but it’s worth explicitly checking what minAvailable it computed:

kubectl get pdb -n kafka

11. Backup and disaster recovery — `deleteClaim: false` is not a backup strategy

§4’s deleteClaim: false prevents a PVC from being deleted when its pod is deleted — genuinely important (losing it accidentally on a routine pod restart would be a real data-loss bug), but it’s easy to over-interpret this as “my data is backed up,” which it isn’t.

What deleteClaim: false actually protects against: a pod being rescheduled, restarted, or the StatefulSet being scaled down and back up — the PVC and its underlying data persist through all of these.
What it does NOT protect against: the underlying storage volume itself failing (a cloud provider’s disk-level failure, however rare), the entire namespace or cluster being accidentally deleted, or a human error that deletes the PVC directly (kubectl delete pvc).
Real backup strategy needs to be separate and explicit — options include MirrorMaker 2 replicating to a second cluster in a different region/cluster entirely (a genuinely different failure domain, not just a different pod), or periodic snapshots of the underlying cloud storage volumes if the storage class supports it (VolumeSnapshot in Kubernetes).

Why this deserves explicit attention rather than assuming RF=3 (Day 10) already covers it: replication factor 3 protects against losing individual brokers within the same cluster — it does nothing for “the whole cluster/namespace is gone” scenarios. deleteClaim: false is a guard against a specific, narrower category of accident (routine pod lifecycle events), not a substitute for genuine cross-cluster or cross-region disaster recovery planning.

12. Common pitfalls

Deploying version: 3.7.0 / inter.broker.protocol.version in a course about Kafka 4 — a real version mismatch worth catching before it propagates into a production manifest; use metadata.version instead, and confirm the Strimzi operator version actually supports Kafka 4.0.x (§8)
Listing “rack awareness” as a feature without configuring rack.topologyKey — Kubernetes’ scheduler has no Kafka-aware reason to spread broker replicas across AZs without it (§9)
Setting JVM -Xmx too close to the container memory limit — starves off-heap needs (network buffers, page cache, RocksDB) and risks OOM-kills that look like unexplained pod crashes rather than an obvious memory misconfiguration (§10)
Treating deleteClaim: false as sufficient disaster recovery — it protects against routine pod lifecycle events, not storage-level failure, accidental namespace deletion, or region-level outages (§11)
Assuming any Strimzi operator version works with any Kafka version — always check the operator’s documented supported-versions table before pinning a Kafka CR’s version field (§2)

Key Takeaways

Strimzi operator manages Kafka lifecycle on K8s — declare CRD, operator does the rest
KRaft mode: annotate Kafka CR with strimzi.io/kraft: enabled — no ZooKeeper pods
KafkaNodePool separates broker (data) and controller (metadata) roles into distinct StatefulSets
For Kafka 4, use metadata.version — inter.broker.protocol.version is a ZooKeeper-era config that KRaft-only Kafka 4 doesn’t use the same way
Rack awareness needs explicit rack.topologyKey configuration — it doesn’t happen automatically just because Strimzi supports it
JBOD persistent volumes survive pod restarts — set deleteClaim: false in production, but pair it with real cross-cluster/region backup, not as a substitute for one
JVM heap should leave real headroom below the container memory limit — off-heap needs (network buffers, page cache, RocksDB) require space too
PodDisruptionBudget ensures rolling updates never drop below minAvailable replicas — Strimzi manages this automatically based on replicas
Strimzi auto-generates and rotates TLS certificates — no manual keytool steps needed
KafkaUser/KafkaTopic CRDs declaratively express the same SCRAM/ACL/topic operations covered manually in Days 44-45 — same underlying model, different interface

Support me through GitHub Sponsors.

Thank you for Reading !! See you in the next post.

➡️ Day 51: Cluster sizing — partitions, replication, disk math

Resources

📘 Kafka: The Definitive Guide — Chapter 2 (Installing Kafka)
🌐 strimzi.io/docs
🌐 strimzi.io/blog/2023/10/kraft-mode
🌐 Strimzi — Supported Kafka versions

Link to Medium blog