60-Day Kafka 4 Learning Plan · Week 4 — Day 25 of 60

---

60-Day Kafka 4 Learning Plan · Week 4 — Schema & Serialization Sources: Kafka: The Definitive Guide Ch.3 · protobuf.dev/programming-guides/proto3 · docs.confluent.io/kafka-protobuf-serializer

Goal

Write a .proto schema for Kafka events, generate type-safe Java code with the Maven plugin, configure KafkaProtobufSerializer in Spring Boot, understand field presence semantics and wire-compatible type changes, and know when to prefer Protobuf over Avro.

1. What is Protobuf?

Protocol Buffers (Protobuf) is Google’s language-neutral binary serialization format. You define schemas in .proto files; protoc generates type-safe code for 20+ languages.

Strengths

• Smallest binary payload of the three formats (JSON / Avro / Protobuf)
• Fastest serialization and deserialization at high throughput
• First-class cross-language code generation

Tradeoffs

• Not human-readable — requires tooling to inspect messages
• Build step required (protoc) — adds complexity to CI
• More verbose than Avro for small payloads

Best for Kafka when

• Teams span multiple languages (Java, Go, Python, Kotlin, Swift…)
• gRPC + Kafka hybrid architecture (same .proto for both)
• Mobile/IoT pipelines where absolute minimum payload size matters

2. Proto3 schema — order_event.proto

// src/main/proto/order_event.proto
syntax = "proto3";

package com.labs.events;
option java_package = "com.labs.events.proto";
option java_outer_classname = "OrderEventProto";

message OrderEvent {
  string order_id    = 1;   // field numbers are permanent — never reuse deleted ones
  string user_id     = 2;
  double amount      = 3;
  Status status      = 4;
  string coupon_code = 5;   // v2 addition — old consumers ignore unknown field 5
}

enum Status {
  PENDING   = 0;
  CONFIRMED = 1;
  FAILED    = 2;
}

Key Proto3 rules

// Mark removed fields as reserved to prevent number reuse
message OrderEvent {
  reserved 6, 7;               // field numbers that must not be reused
  reserved "legacy_field";     // field names that must not be reused
}

3. Code generation — protobuf-maven-plugin

<!-- pom.xml -->
<build>
  <plugins>
    <plugin>
      <groupId>io.github.ascopes</groupId>
      <artifactId>protobuf-maven-plugin</artifactId>
      <version>2.6.0</version>
      <configuration>
        <protocVersion>4.26.1</protocVersion>
      </configuration>
      <executions>
        <execution>
          <goals><goal>generate</goal></goals>
        </execution>
      </executions>
    </plugin>
  </plugins>
</build>

<dependencies>
  <!-- Protobuf Java runtime -->
  <dependency>
    <groupId>com.google.protobuf</groupId>
    <artifactId>protobuf-java</artifactId>
    <version>4.26.1</version>
  </dependency>

  <!-- Confluent Protobuf serializer (requires Confluent Maven repo) -->
  <dependency>
    <groupId>io.confluent</groupId>
    <artifactId>kafka-protobuf-serializer</artifactId>
    <version>7.6.1</version>
  </dependency>
</dependencies>

Running mvn compile generates OrderEventProto.java with OrderEvent, OrderEvent.Builder, and Status enum under target/generated-sources/protobuf/.

4. Spring Boot — KafkaProtobufSerializer config

# application.yml (labs-api producer)
spring:
  kafka:
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: io.confluent.kafka.serializers.protobuf.KafkaProtobufSerializer
      acks: all
    consumer:
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: io.confluent.kafka.serializers.protobuf.KafkaProtobufDeserializer
    properties:
      schema.registry.url: http://localhost:8081
      # Deserialize to specific generated class (not DynamicMessage)
      specific.protobuf.value.type: com.labs.events.proto.OrderEventProto$OrderEvent

5. Producer & consumer with generated class

Producer

@Service
@RequiredArgsConstructor
public class EventProducer {

    private final KafkaTemplate<String, OrderEvent> kafka;

    public void send(String orderId, double amount) {
        // Builder pattern generated by protoc
        var event = OrderEvent.newBuilder()
            .setOrderId(orderId)
            .setUserId("user-123")
            .setAmount(amount)
            .setStatus(Status.CONFIRMED)
            .build();

        kafka.send("labs.events", orderId, event)
             .whenComplete((result, ex) -> {
                 if (ex != null) log.error("Send failed", ex);
             });
    }
}

Consumer

@Component
@Slf4j
public class EventConsumer {

    @KafkaListener(topics = "labs.events", groupId = "socket-service")
    public void consume(ConsumerRecord<String, OrderEvent> record) {
        OrderEvent event = record.value();
        log.info("orderId={} amount={} status={}",
            event.getOrderId(), event.getAmount(), event.getStatus());
    }
}

6. Avro vs Protobuf — choose wisely

Rule of thumb:

• Pure Kafka microservices team → Avro (simpler setup, native SR support)
• gRPC + Kafka, mobile, or multi-language polyglot → Protobuf

7. Field presence — the optional keyword gotcha

The table in §2 says “missing fields use zero-value defaults” — but this hides an important distinction: by default, Proto3 cannot tell you whether a field was explicitly set to its zero value or never set at all.

message OrderEvent {
  double amount = 3;   // amount=0 could mean "explicitly zero" OR "field never set" — indistinguishable
}

OrderEvent event = OrderEvent.parseFrom(bytes);
event.getAmount();  // returns 0.0 either way — no hasAmount() method generated for this field

The fix — explicit optional keyword (added back to proto3 in later versions): generates a presence-tracking hasXxx() method, restoring the ability to distinguish “set to zero” from “not set.”

message OrderEvent {
  optional double amount = 3;   // now tracks presence explicitly
}

if (event.hasAmount()) {
    // field was explicitly set, even if the value happens to be 0.0
}

Why this matters for real data: for a field like discountAmount where 0 is a legitimate, meaningful business value, not knowing whether it was set or defaulted is a genuine correctness gap — e.g. “no discount was ever calculated” vs “a $0 discount was explicitly applied” are different business facts that plain (non-optional) proto3 fields can’t distinguish. Use optional on any numeric/boolean/string field where the zero value is a meaningful, distinct case from absence.

8. oneof and well-known types

oneof — mutually exclusive fields

message PaymentEvent {
  string order_id = 1;
  oneof payment_method {
    CreditCard credit_card = 2;
    BankTransfer bank_transfer = 3;
    string wallet_id = 4;
  }
}

Only one field in a oneof group can be set at a time — setting a second one clears the first. This models “exactly one of these variants” more precisely than a set of independent optional fields, and avoids the awkwardness of a Java class with multiple mutually-exclusive nullable fields.

Well-known types — don’t reinvent timestamps

import "google/protobuf/timestamp.proto";

message OrderEvent {
  google.protobuf.Timestamp created_at = 6;
}

Instant createdAt = Instant.ofEpochSecond(
    event.getCreatedAt().getSeconds(), event.getCreatedAt().getNanos());

Prefer google.protobuf.Timestamp over a raw int64 epoch-millis field — same reasoning as Avro’s logical types (Day 22 §8): it’s self-documenting in the schema, has standard conversion helpers in every language’s generated code, and avoids every team inventing its own “is this millis or seconds” convention.

9. Wire-compatible type changes — more lenient than Avro in specific ways

Protobuf’s wire format groups types into compatibility classes — changing within a class is safe even though the Java type changes, which is more permissive than Avro’s exact-type-match rule (Day 24 §3).

NOT safe: changing between wire type groups — e.g. int32 (varint) → fixed32 (fixed-width) is a breaking change even though both are “numbers,” because the byte layout on the wire differs.

Practical caution: just because the wire format tolerates a change doesn’t mean it’s free of application-level risk — string → bytes compiles and deserializes fine, but every consumer’s business logic that assumed a UTF-8 string now needs review. Treat wire-compatibility as “won’t crash on deserialize,” not “safe to change without checking downstream code.”

10. Testing Protobuf serialization

class OrderEventProtobufTest {

    @Test
    void roundTripsWithoutDataLoss() {
        OrderEvent original = OrderEvent.newBuilder()
            .setOrderId("ord-1").setUserId("user-1").setAmount(99.99)
            .setStatus(Status.CONFIRMED).build();

        byte[] bytes = original.toByteArray();
        OrderEvent deserialized = OrderEvent.parseFrom(bytes);

        assertThat(deserialized).isEqualTo(original);
    }

    @Test
    void oldMessageWithoutNewFieldParsesWithDefaultInNewCode() throws InvalidProtocolBufferException {
        // Simulates a v1-encoded message being read by v2 generated code
        // that added coupon_code — should parse without error, coupon_code == ""
        byte[] v1Bytes = buildV1MessageBytes(); // no field 5 present
        OrderEvent parsed = OrderEvent.parseFrom(v1Bytes);
        assertThat(parsed.getCouponCode()).isEmpty(); // zero-value default, not an exception
    }
}

As with Avro (Day 22 §11), the cross-version test is the one that actually catches regressions — a same-version round-trip test always passes and doesn’t tell you anything about whether last month’s messages still parse correctly with today’s generated code.

11. Common pitfalls

• Reusing a deleted field number — the single most dangerous Protobuf mistake; old and new messages silently misinterpret each other’s bytes for that field number. Always use reserved instead of just deleting a field declaration.
• Not using optional for fields where the zero value is meaningful — silently loses the ability to distinguish “explicitly zero” from “never set” (§7)
• Treating wire-compatible type changes as fully free — string ↔ bytes won’t crash deserialization, but downstream code assuming UTF-8 text needs review anyway (§9)
• Reinventing timestamp/duration encoding instead of using google.protobuf.Timestamp/Duration — loses cross-language convention and standard conversion helpers (§8)
• Assuming Protobuf’s Schema Registry compatibility checking is identical to Avro’s — the underlying rules differ (field-number-based vs name-based resolution); don’t port Avro compatibility assumptions over without verifying against Protobuf’s actual rules (Day 26 covers this in depth)

Key Takeaways

• Proto3 defines messages with typed, numbered fields — field numbers are permanent
• protoc (via Maven plugin) generates type-safe Java builders from .proto at compile time
• KafkaProtobufSerializer/Deserializer from Confluent integrate with Schema Registry
• Never reuse a deleted field number — use reserved to prevent accidental reuse
• All Proto3 fields are optional by default — no required keyword exists in Proto3, but the explicit optional keyword restores presence tracking (hasXxx()) where the zero value is meaningful
• oneof models mutually exclusive fields cleanly; well-known types like Timestamp avoid reinventing conventions
• Protobuf’s wire-compatible type groups (e.g. int32↔int64) are more lenient than Avro’s exact-type rule — but wire-safety isn’t the same as application-safety
• Choose Avro for Kafka-first teams; Protobuf for cross-language or gRPC + Kafka architectures

---

Support me through GitHub Sponsors.

Thank you for Reading !! See you in the next post.

Next

➡️ Day 26: Compatibility modes — BACKWARD, FORWARD, FULL Deep Dive

Resources

• 📘 Kafka: The Definitive Guide — Chapter 3 (Serializers)
• 🌐 protobuf.dev/programming-guides/proto3
• 🌐 docs.confluent.io — Protobuf Serializer
• 🌐 Protocol Buffers — Well-Known Types

👉 Link to Medium blog