60-Day Kafka 4 Learning Plan · Week 4 — Day 22 of 60

60-Day Kafka 4 Learning Plan · Week 4 — Schema & Serialization Sources: Kafka: The Definitive Guide Ch.3 · avro.apache.org/docs · confluent.io/schema-registry

Goal

Understand why JSON serialization becomes a liability at Kafka production scale, how Apache Avro binary encoding solves it, the difference between generated and generic record access, schema evolution constraints that trip teams up, and when to use JSON, Avro, or Protobuf.

1. JSON pain points at scale

❌ No schema enforcement — any producer can write any shape; consumers crash silently on unexpected fields or missing keys
❌ Field names repeated in every message — "orderId" repeated 1 million times = 7 MB of wasted bytes per topic per second
❌ No type safety — "amount": "99.99" and "amount": 99.99 are both valid JSON but will break a consumer expecting a double
❌ Schema evolution is manual — adding or removing a field breaks consumers with no warning; no compatibility check at publish time
❌ Slow parse — text parsing is 3–10× slower than binary deserialization under high-throughput load

2. Same event — JSON vs Avro bytes

JSON payload (~138 bytes)

{
  "orderId": "ord-abc-123",
  "userId": "user-xyz-456",
  "amount": 99.99,
  "status": "CONFIRMED"
}

Avro binary payload (~42 bytes) ★ — 70% smaller

No field names are present in the payload. The schema is identified by a 4-byte ID looked up in Schema Registry. Values are packed as binary:

Strings: varint-encoded length prefix + UTF-8 bytes
Doubles: 8 bytes IEEE 754
Enums: integer index into the symbol list

Wire format: [0x00 magic][schema-id 4B][avro binary data]

At 1M messages/second: Avro saves ~96 MB/s in network bandwidth compared to JSON — a significant difference in cloud egress costs and broker disk I/O.

3. Avro schema — the data contract

A .avsc file (JSON schema definition) acts as the shared contract between producer and consumer. Both compile it at build time — schema mismatch becomes a compilation error, not a runtime crash.

{
  "type": "record",
  "name": "OrderEvent",
  "namespace": "com.labs.events",
  "fields": [
    {"name": "orderId", "type": "string"},
    {"name": "userId",  "type": "string"},
    {"name": "amount",  "type": "double"},
    {"name": "status",  "type": {
      "type": "enum",
      "name": "Status",
      "symbols": ["PENDING", "CONFIRMED", "FAILED"]
    }},
    {"name": "createdAt", "type": "long", "logicalType": "timestamp-millis",
     "default": 0}
  ]
}

Avro primitive types

Optional fields (nullable with default)

{"name": "couponCode", "type": ["null", "string"], "default": null}

Union ordering matters: ["null", "string"] with "default": null is correct; writing ["string", "null"] changes the Avro default-value semantics and is rejected by most schema evolution tooling. Always put "null" first in a union when the field is meant to be optional.

4. JSON vs Avro vs Protobuf at a glance

5. When to use what

JSON

Development, debugging, low-volume topics
REST APIs and config files where human readability is the priority
Small teams where schema tooling overhead isn’t worth it yet

Avro ★ (recommended for Kafka production)

Kafka production pipelines at any meaningful scale
Schema Registry workflows with compatibility enforcement
Event-driven microservices where the schema is the API contract

Protobuf

Cross-language gRPC services (Java ↔ Go ↔ Python)
Mobile/IoT where absolute minimum payload size matters
Teams that prefer code generation from .proto files over Avro’s JSON schema

6. How Schema Registry fits in

Producer (labs-api)
  → serialize OrderEvent with Avro
  → register schema in Schema Registry (first time only)
  → get schema_id back
  → write [0x00][schema_id][avro bytes] to Kafka

Consumer (labs-socket)
  → read message from Kafka
  → extract schema_id from first 5 bytes
  → fetch schema from Schema Registry (cached after first fetch)
  → deserialize avro bytes → OrderEvent POJO

Schema Registry prevents schema-incompatible producers from publishing — compatibility is enforced at registration time, not at consumption time. (Full setup in Day 23.)

7. Specific records vs generic records

Avro gives you two ways to work with a deserialized message, and the choice affects both developer experience and coupling.

<!-- avro-maven-plugin generates OrderEvent.java from src/main/avro/OrderEvent.avsc -->
<plugin>
    <groupId>org.apache.avro</groupId>
    <artifactId>avro-maven-plugin</artifactId>
    <version>1.11.3</version>
    <executions>
        <execution>
            <phase>generate-sources</phase>
            <goals><goal>schema</goal></goals>
            <configuration>
                <sourceDirectory>${project.basedir}/src/main/avro</sourceDirectory>
                <outputDirectory>${project.basedir}/src/main/java</outputDirectory>
            </configuration>
        </execution>
    </executions>
</plugin>

// GenericRecord usage — no generated class needed, useful for a generic DLT replayer
GenericRecord record = (GenericRecord) deserialized;
String orderId = record.get("orderId").toString();

Recommendation: default to SpecificRecord for services that own and understand a schema — it catches field-name typos at compile time instead of a runtime ClassCastException. Reach for GenericRecord only for genuinely schema-agnostic infrastructure code.

8. Logical types — beyond the primitives

Logical types layer semantic meaning on top of a primitive Avro type, so 1721520000000 in the wire format is understood as a timestamp, not just a long.

{"name": "orderTotal", "type": "bytes", "logicalType": "decimal", "precision": 10, "scale": 2}

Why decimal matters for money fields: using Avro’s native double for currency amounts introduces floating-point rounding error over repeated arithmetic — exactly the kind of bug that’s invisible in testing and painful in a financial reconciliation. Use the decimal logical type (backed by BigDecimal in Java) for any monetary field, not double.

9. Enum evolution — a common schema-evolution trap

Unlike adding an optional field (safe, covered in Day 24), removing or reordering enum symbols is a breaking change that Avro’s compatibility checking treats differently than you might expect.

// v1
{"type": "enum", "name": "Status", "symbols": ["PENDING", "CONFIRMED", "FAILED"]}

// v2 — adding a new symbol is SAFE (backward compatible)
{"type": "enum", "name": "Status", "symbols": ["PENDING", "CONFIRMED", "FAILED", "REFUNDED"]}

// v2 — removing a symbol is UNSAFE — old messages encoding that symbol's index
// can no longer be decoded correctly by a reader using the new schema
{"type": "enum", "name": "Status", "symbols": ["PENDING", "CONFIRMED"]}  // ⚠ FAILED removed

Why reordering is also dangerous: older Avro encodings can store an enum as its integer index into the symbols list, so symbols: ["A", "B", "C"] → ["B", "A", "C"] can silently flip the meaning of previously-written data under some encoding/tooling combinations. Treat enum symbol lists as append-only — new values go at the end, existing values are never removed or reordered.

10. Migrating an existing JSON topic to Avro

You can’t atomically flip a live topic’s serialization format — consumers mid-deployment would break. The safe path mirrors the dual-publish pattern from Day 19, applied to serialization instead of brokers:

New topic, not in-place conversion. Create orders-v2 with the Avro schema; keep orders (JSON) running unchanged.
Dual-write from the producer: JSON to orders, Avro to orders-v2, for a transition window.
Migrate consumers one at a time to orders-v2, same incremental approach as Day 19 §5.
Deprecate orders once all consumers have moved, then decommission.

Why not convert in place: Kafka has no way to retroactively re-encode already-written JSON records as Avro, and even if it could, any consumer still mid-rollout to the new deserializer would crash on encountering old-format bytes. A parallel topic sidesteps this entirely and gives you the same safe, incremental rollback capability as any other high-stakes migration.

11. Testing serialization round-trip

class OrderEventAvroTest {

    @Test
    void roundTripsThroughAvroWithoutDataLoss() throws IOException {
        OrderEvent original = new OrderEvent("ord-1", "user-1", 99.99, Status.CONFIRMED, Instant.now());

        DatumWriter<OrderEvent> writer = new SpecificDatumWriter<>(OrderEvent.class);
        ByteArrayOutputStream out = new ByteArrayOutputStream();
        Encoder encoder = EncoderFactory.get().binaryEncoder(out, null);
        writer.write(original, encoder);
        encoder.flush();

        DatumReader<OrderEvent> reader = new SpecificDatumReader<>(OrderEvent.class);
        Decoder decoder = DecoderFactory.get().binaryDecoder(out.toByteArray(), null);
        OrderEvent deserialized = reader.read(null, decoder);

        assertThat(deserialized).isEqualTo(original); // exact equality — no precision/type loss
    }

    @Test
    void oldSchemaMessageReadableByNewSchemaReader() throws IOException {
        // Write with a schema missing a field added later (e.g. couponCode);
        // read with the current schema — the new field should resolve to its default,
        // not throw. This is the real regression to guard against on every schema change.
    }
}

The second test matters more than the first in practice — a plain round-trip test always passes; what actually breaks production is an old message being read by a newer schema (or vice versa) after a deploy. Test cross-version compatibility explicitly, not just same-version serialization.

12. Common pitfalls

Writing ["string", "null"] instead of ["null", "string"] for optional fields — breaks default-value semantics that most tooling expects
Removing or reordering enum symbols — treat symbol lists as append-only; see §9
Using double for monetary fields — use the decimal logical type to avoid floating-point rounding error in financial data
Choosing GenericRecord by default “to keep things flexible” — loses compile-time safety for no benefit in ordinary application code; reserve it for genuinely schema-agnostic infrastructure
Attempting to convert an existing topic’s format in place — always migrate via a parallel topic (§10), never assume old and new format bytes can coexist safely on one topic

Key Takeaways

JSON has no schema enforcement — any shape can break a consumer silently
Avro binary is ~70% smaller than JSON by omitting field names from every message
Avro schema is a contract — producer and consumer must agree or fail at registration
Schema Registry stores schemas centrally — producers register, consumers look up by ID
Avro supports backward/forward schema evolution — add optional fields safely (Day 24)
Default to SpecificRecord (generated POJOs) for application code; reserve GenericRecord for schema-agnostic infrastructure
Use the decimal logical type for money — never double
Enum symbol lists are append-only — removing or reordering symbols breaks compatibility with existing data
Migrating a live JSON topic to Avro means a parallel topic + incremental consumer cutover, never an in-place format change
Use JSON for dev/debug; migrate to Avro before hitting production scale

Support me through GitHub Sponsors.

Thank you for Reading !! See you in the next story.

➡️ Day 23: Schema Registry — Confluent setup & Spring integration

Resources

📘 Kafka: The Definitive Guide — Chapter 3 (Producers: Serializers)
🌐 avro.apache.org/docs
🌐 docs.confluent.io/platform/current/schema-registry

👉 Link to Medium blog