Kafka

60-Day Kafka 4 Learning Plan · Week 9 — Day 60 of 60

Editorial Team

60-Day Kafka 4 Learning Plan · Week 9 — Capstone & Career · FINAL DAY The series is complete. Here is where to go next.

Flink reads from Kafka, processes with sub-second latency, and writes back to Kafka or a database. It is more powerful than Kafka Streams for complex event patterns, ML feature computation, global aggregations across multiple streams, and long-running stateful jobs.

// Flink DataStream API — tumbling window count per user
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

KafkaSource<EventDto> source = KafkaSource.<EventDto>builder()
    .setBootstrapServers("broker:9092")
    .setTopics("labs.events")
    .setGroupId("flink-group")
    .setStartingOffsets(OffsetsInitializer.earliest())
    .setValueOnlyDeserializer(new JsonDeserializationSchema<>(EventDto.class))
    .build();

env.fromSource(source, WatermarkStrategy.noWatermarks(), "Kafka")
   .keyBy(EventDto::userId)
   .window(TumblingEventTimeWindows.of(Duration.ofMinutes(5)))
   .aggregate(new CountAggregator())
   .sinkTo(KafkaSink.<UserCount>builder()
       .setBootstrapServers("broker:9092")
       .setRecordSerializer(KafkaRecordSerializationSchema.builder()
           .setTopic("labs.user-counts")
           .setValueSerializationSchema(new JsonSerializationSchema<>())
           .build())
       .build());

env.execute("User Event Count Job");

When to choose Flink over Kafka Streams:

  • Complex event processing across multiple streams
  • ML model scoring on streaming data
  • Sub-second latency requirements with large state
  • Global aggregations that don’t fit in one JVM

2. Apache Pulsar — Kafka’s closest alternative

A note on the “community size” row: the original material cited a specific contributor count here — that kind of precise number changes constantly and is easy to state with false confidence. The durable, verifiable fact is the comparative one: Kafka’s contributor base and surrounding ecosystem (client libraries, connectors, managed offerings) is substantially larger and more mature than Pulsar’s, which is what actually matters for this comparison — not a specific headcount that will be stale by the time anyone reads it.

Bottom line: Kafka 4 remains the industry standard. Learn Pulsar when you encounter a use case that needs storage/compute separation (e.g., a cloud provider building multi-tenant messaging-as-a-service).

3. Kafka ecosystem tools — know these for interviews

Conduktor

A commercial Kafka management platform. Beyond just browsing topics, it adds: data masking policies, consumer group management, schema governance, audit logs, and alerting — all with a team-friendly UI. Free tier available.

# Run Conduktor locally (Docker)
docker run -p 8080:8080 \
  -e BOOTSTRAP_SERVERS=broker:9092 \
  conduktor/conduktor-console:latest

Kafka UI (open source)

Free alternative to Conduktor. Browse topics, produce test messages, inspect consumer groups, manage Schema Registry — all in one browser-based UI.

# docker-compose.yml — add Kafka UI to your stack
kafka-ui:
  image: provectuslabs/kafka-ui:latest
  ports: ["8080:8080"]
  environment:
    KAFKA_CLUSTERS_0_NAME: local
    KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: broker:9092
    KAFKA_CLUSTERS_0_SCHEMAREGISTRY: http://schema-registry:8081

Cruise Control (LinkedIn)

Automatically rebalances Kafka clusters — moves partitions across brokers to optimise resource utilisation, handle broker additions, and prepare for rolling restarts.

# Trigger a full cluster rebalance
curl -X POST "http://cruise-control:9090/kafkacruisecontrol/rebalance?dryrun=false"

Debezium

Change Data Capture (CDC) connector. Streams INSERT/UPDATE/DELETE events from PostgreSQL, MySQL, MongoDB, and others directly into Kafka topics via the database’s write-ahead log — no polling.

// Debezium PostgreSQL connector config
{
  "name": "labs-pg-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgres",
    "database.dbname": "labsdb",
    "table.include.list": "public.orders",
    "topic.prefix": "labs",
    "plugin.name": "pgoutput"
  }
}

KEDA (Kubernetes Event-Driven Autoscaler)

Scales Kubernetes pods based on Kafka consumer lag. The missing HPA for Kafka — scales consumers up when lag grows and back down when it’s cleared. (Used in Day 56 capstone.)

Schema Registry

Confluent (self-hosted or cloud) or AWS Glue Schema Registry. Stores Avro/Protobuf/JSON schemas, enforces compatibility rules, and provides the schema ID used in binary serialisation.

4. Your next learning steps

5. What you built in 60 days

✅ Kafka 4 KRaft cluster — no ZooKeeper, full local + Kubernetes setup
✅ Spring Boot 3 labs-api + labs-socket — producer, consumer, DLT, retry
✅ Avro + Schema Registry — schema evolution, BACKWARD compatibility
✅ Kafka Streams 4 DSL — KStream, KTable, tumbling windows, state stores
✅ Kafka Connect pipeline — DB → Kafka → Redis → WebSocket
✅ ksqlDB — CSAS, windowed aggregations, push/pull queries
✅ Full security stack — TLS + SCRAM-SHA-512 + ACLs + mTLS
✅ Prometheus + Grafana — 6 dashboard panels, alert rules, Slack webhooks
✅ Strimzi on Kubernetes — KafkaNodePool, PVC, KEDA HPA
✅ MirrorMaker 2 — active-passive DR, offset translation
✅ Disaster recovery runbooks — RTO/RPO, broker failure, cluster failover
✅ Capstone: REST → Kafka → WebSocket → Flutter — < 200ms end-to-end

🏆 60 infographics · 60 GitHub markdown docs · 60 days of Kafka 4 mastery

6. Share Groups (Queues for Kafka) — a real Kafka 4.x feature this course never covered

Worth naming explicitly on the final day, precisely because it’s genuinely new and genuinely relevant to everything covered across these 60 days: Kafka introduced share groups (KIP-932, “Queues for Kafka”) as an alternative to the consumer group model this entire course was built around.

The core difference from everything in Days 6, 17, 29, 56:

Consumer groups (everything covered in this course):
  Each partition is assigned EXCLUSIVELY to one consumer in the group
  → max useful parallelism = partition count (the rule from Day 6 §1,
    repeated across Day 17, Day 29, Day 51...)

Share groups (new):
  Multiple consumers can cooperatively process records from the SAME
  partition, with individual per-message acknowledgment and delivery
  tracking — closer to a traditional queue's work-distribution model
  → consumer count can EXCEED partition count, unlike everything
    this course taught about the partition-count ceiling

Why this doesn’t invalidate anything you learned across the last 60 days: consumer groups remain the right model for the vast majority of use cases in this course — anything needing per-key ordering (Day 12), exactly-once semantics (Day 11), or Kafka Streams’ partition-aligned state stores (Day 29) still fundamentally depends on the exclusive-partition-assignment model. Share groups exist specifically for queue-like work distribution where strict per-partition ordering isn’t the goal and you want RabbitMQ/SQS-style flexible worker scaling instead — genuinely useful for a subset of workloads, not a replacement for everything covered in Weeks 1–8. Worth knowing it exists and roughly what problem it solves, as the kind of “what’s actually new since I learned the fundamentals” awareness that keeps knowledge current after a structured course ends — which is the whole point of this final day.

7. Staying current after this course ends

The single most durable meta-lesson from a 60-day technical course is knowing how to stay current once the structured content stops — Kafka itself doesn’t stop evolving on Day 60, and this course’s own material has already needed several corrections during enrichment (day-numbering fixes, a factual error about min.insync.replicas behavior that appeared twice, stale managed-service version claims) purely from things changing or being stated imprecisely along the way.

  • Track KIPs (Kafka Improvement Proposals) at cwiki.apache.org — this is where features like share groups (§6), tiered storage (Day 9 §7, KIP-405), and the exactly-once transactional model (Day 11, KIP-98) were originally proposed and designed; reading new KIPs as they land is the most direct way to know what’s coming before it shows up in a blog post.
  • Re-verify version-specific claims before trusting them, including from this course — exactly the discipline applied throughout this enrichment (Day 50’s Kafka 3.7→4.0 correction, Day 53’s stale managed-service version table) is the same discipline worth applying to any future source, including official-looking documentation that may itself lag behind the latest release.
  • Revisit the recurring pattern list from Day 57 §9 periodically — the specific technologies will keep changing, but “check replication factor,” “check for auth on any new admin surface,” “distrust absolute alert thresholds,” “ask what happens with 2+ instances,” and “a config that looks right isn’t a tested plan” will keep applying to whatever comes next, including Flink, Pulsar, or share groups themselves.

Final words

Kafka 4 with KRaft is the most significant architectural shift in Kafka’s history. You now understand not just the API — but the internals, the operations, the security model, the cloud options, and the production patterns that separate senior engineers from the rest.

Keep building. Keep shipping. The event streaming world needs engineers like you.

Support me through GitHub Sponsors.

Thank you for Reading !! See you in the next post.

Resources

, ,

Related Posts

Kafka

60-Day Kafka 4 Learning Plan · Week 9 — Day 59 of 60

Editorial Team
Flutter Kafka Spring Boot

60-Day Kafka 4 Learning Plan · Week 9 — Day 58 of 60

Editorial Team