60-Day Kafka 4 Learning Plan · Week 4 — Day 24 of 60

60-Day Kafka 4 Learning Plan · Week 4 — Schema & Serialization Sources: Kafka: The Definitive Guide Ch.9 · avro.apache.org/docs/current/spec

Goal

Understand Avro’s three compatibility modes, know exactly which schema changes are safe vs breaking — including for nested types and unions — how to sequence a rolling deployment safely, and how to choose and enforce the right mode in CI.

1. Why schema evolution matters

Kafka retains messages for days or weeks. When you deploy a new schema version, old messages with the old schema and new messages with the new schema coexist in the same topic. Every consumer must be able to read both.

⚠️ The problem: Producer adds a new required field "couponCode" and deploys. Old consumer reads a message written before the change — couponCode is absent. Does it crash? Schema Registry’s compatibility mode decides at registration time.

2. The three compatibility modes

BACKWARD ★ (default — recommended)

New schema can read data written by old schema.

Safe to do: add fields with default values; remove optional fields
Deploy order: upgrade consumers first, then producers
Why: consumers with the new schema can read old messages (field absent → use default)

FORWARD

Old schema can read data written by new schema.

Safe to do: add fields (old consumer ignores unknown); remove fields that have defaults
Deploy order: upgrade producers first, then consumers
Why: old consumers reading new messages can skip unknown fields

FULL

Both BACKWARD and FORWARD at the same time.

Safe to do: only add or remove fields that have default values
Deploy order: any order is safe — most conservative mode
Why: every version can read every other version’s data

3. Safe changes vs breaking changes

The golden rule: always add new fields with a default value. This is safe under all three modes.

4. Safe evolution — adding an optional field

v1 — order-event.avsc

{
  "type": "record",
  "name": "OrderEvent",
  "namespace": "com.labs.events",
  "fields": [
    {"name": "orderId", "type": "string"},
    {"name": "userId",  "type": "string"},
    {"name": "amount",  "type": "double"}
  ]
}

v2 — add couponCode (BACKWARD compatible ✅)

{
  "type": "record",
  "name": "OrderEvent",
  "namespace": "com.labs.events",
  "fields": [
    {"name": "orderId",    "type": "string"},
    {"name": "userId",     "type": "string"},
    {"name": "amount",     "type": "double"},
    {"name": "couponCode", "type": ["null", "string"], "default": null}
  ]
}

Old consumer reading v2 message: couponCode missing from payload → consumer uses null default. No crash. ✅
New consumer reading v1 message: couponCode absent in payload → consumer uses null default. No crash. ✅

Renaming a field — use aliases (not a type change)

{
  "name": "orderIdentifier",
  "type": "string",
  "aliases": ["orderId"]
}

Avro aliases let the new schema resolve the old field name during deserialization — the only safe way to rename a field.

5. Set compatibility mode in Registry

Set global default

curl -X PUT http://localhost:8081/config \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"compatibility": "BACKWARD"}'

Set per-subject override

curl -X PUT http://localhost:8081/config/labs.events-value \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"compatibility": "FULL"}'

Check current compatibility for a subject

curl http://localhost:8081/config/labs.events-value
# → {"compatibilityLevel":"FULL"}

Test schema compatibility before registering

curl -X POST http://localhost:8081/compatibility/subjects/labs.events-value/versions/latest \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"schema": "{ ...new schema json... }"}'
# → {"is_compatible": true}

Use this in your CI/CD pipeline (Day 27) to catch breaking changes before deployment.

6. Additional compatibility levels

Use _TRANSITIVE variants when you need a new schema to be compatible with all historical versions, not just the most recent one.

BACKWARD (non-transitive) has a gap worth knowing: it only checks against the immediately preceding version. If v1 → v2 was compatible and v2 → v3 was compatible, that does not guarantee v1 → v3 is compatible — a field removed in v2 and a different field removed in v3 can combine into a v1 message v3 can’t read. BACKWARD_TRANSITIVE closes this gap by checking against every historical version, at the cost of being more restrictive about what changes are allowed. For topics with long retention where very old messages might still be read, prefer the transitive variant.

7. Evolution rules for nested types

Real schemas usually aren’t flat — arrays, maps, and nested records follow the same core principles (defaults are safe, type changes aren’t) but have their own specific rules.

Arrays and maps

// Adding an item type change is breaking, same as any other type change
{"name": "tags", "type": {"type": "array", "items": "string"}}
// v2: {"type": {"type": "array", "items": "int"}}  ❌ breaking — item type changed

Arrays/maps themselves don’t need defaults when added as a new field — an empty array/map is generally the implicit safe default, but explicitly specifying "default": [] (or {} for maps) is still the clearer, safer choice.

Nested records

{"name": "shippingAddress", "type": {
  "type": "record", "name": "Address",
  "fields": [
    {"name": "street", "type": "string"},
    {"name": "city", "type": "string"}
  ]
}}

Fields within a nested record follow exactly the same rules as top-level fields — adding a field to Address needs a default, just like adding one to OrderEvent directly. Compatibility checking recurses into nested records; a breaking change buried three levels deep still fails the whole schema’s compatibility check.

Practical implication: a schema with deeply nested records means a compatibility break can hide anywhere in the tree — this is a good reason to prefer flatter schemas where practical, and to lean on the CI compatibility check (§5, §11) rather than manual review for anything beyond a shallow schema.

8. Union evolution rules

Unions (["null", "string"] and similar) have evolution rules distinct from simple field changes.

// v1
{"name": "discount", "type": ["null", "double"], "default": null}

// v2 — adding int support, e.g. supporting percentage-as-int discounts
{"name": "discount", "type": ["null", "double", "int"], "default": null}

The default value’s type must match the union’s first member. If "default": null, the union must list "null" first — this is the same rule from Day 22 §3, worth repeating here because it’s specifically a schema-evolution failure mode: a schema that looks fine in isolation can fail Registry validation purely because of default/union-order mismatch.

9. Deploy sequencing — a worked rolling-deployment timeline

Choosing BACKWARD vs FORWARD isn’t just a Registry setting — it dictates the order you’re allowed to deploy producer and consumer changes safely. Here’s what BACKWARD compatibility actually looks like across a real rolling deployment:

T0: v1 schema live. Producers (labs-api) and consumers (labs-socket) both on v1.

T1: Deploy NEW consumer code (labs-socket) that understands v2 schema
    (couponCode field, with default). Consumer is now v2-aware.
    → Still reading v1 messages fine — couponCode resolves to its default.

T2: Deploy NEW producer code (labs-api) that starts writing v2 messages
    (includes couponCode). Registry validates v2 against BACKWARD compatibility
    with v1 — passes, since couponCode has a default.
    → New messages now have couponCode populated.

T3: Both services fully on v2. Old v1 messages still in the topic (within
    retention) remain readable by any consumer instance, old or new.

Why consumers deploy first under BACKWARD: if producers deployed first (writing v2 messages) while old consumer instances (still v1-only, no knowledge of couponCode) were still running during the rollout, those old consumers would simply ignore the unknown field — which is actually fine under BACKWARD too, since old code reading a superset of known fields doesn’t break. The real risk BACKWARD protects against is the reverse rollout order for FORWARD-only changes; get the deploy order matched to your compatibility mode, and don’t assume “consumer first” is universally required regardless of mode — it’s specifically what BACKWARD’s guarantee is built around.

10. Choosing a compatibility mode — decision guide

Default to BACKWARD unless you have a specific reason not to. It matches the most common real-world deployment pattern (deploy consumers ahead of producers) and is what most Kafka/Avro tooling assumes when documentation doesn’t specify otherwise.

11. Testing schema evolution in CI

This is where the compatibility check from §5 and the CI pipeline pattern from Day 23 §10 come together — the pipeline step should test the actual compatibility mode configured for that subject, not just “does the schema parse.”

@Test
void newSchemaVersionIsBackwardCompatible() throws IOException {
    Schema oldSchema = new Schema.Parser().parse(new File("src/test/resources/order-event-v1.avsc"));
    Schema newSchema = new Schema.Parser().parse(new File("src/main/avro/order-event.avsc"));

    SchemaValidator validator = new SchemaValidatorBuilder().canReadStrategy().validateLatest();
    // Throws SchemaValidationException if newSchema cannot read data written with oldSchema
    validator.validate(newSchema, List.of(oldSchema));
}

Why test this locally too, not just via the Registry’s REST compatibility endpoint: a local SchemaValidator check runs in seconds as part of the normal test suite, catching an obviously broken change (e.g. a field type flip) before even reaching CI’s Registry-dependent step — cheaper feedback loop, same underlying Avro resolution rules.

12. Common pitfalls

Assuming non-transitive BACKWARD protects against all historical versions — it only checks the immediately preceding one; use _TRANSITIVE for long-retention topics (§6)
Union type/default mismatch — "default": null requires "null" to be the first union member; this specific failure mode shows up exactly when you’re trying to make an evolution “safe” (§8)
Assuming array/map item type changes are as safe as adding a field — they’re not; an item type change is a breaking type change like any other (§7)
Deploying in the wrong order for the configured mode — BACKWARD assumes consumers-first, FORWARD assumes producers-first; mixing these up defeats the compatibility guarantee even though the schema change itself was “safe” on paper (§9)
Picking NONE for convenience during prototyping and forgetting to tighten it — fine with zero consumers, a real production risk the moment even one consumer starts depending on the topic

Key Takeaways

BACKWARD (default): new schema reads old data — upgrade consumers first
FORWARD: old schema reads new data — upgrade producers first
FULL: both directions — safest, any upgrade order is safe
Always add new fields with a default value — never add required fields
Renaming or retyping a field is always breaking — use Avro aliases to rename safely
Non-transitive modes only check against the immediately preceding version — use _TRANSITIVE variants for long-retention topics
Nested records recurse the same compatibility rules; a break can hide several levels deep in a complex schema
Union evolution can add types safely but never remove them, and default/union-order mismatches are a common failure mode
Deploy order must match the configured compatibility mode — BACKWARD assumes consumers-first, FORWARD assumes producers-first
Schema Registry enforces compatibility at registration time — incompatible schemas are rejected — but a local SchemaValidator test in CI catches obvious breaks even faster

Support me through GitHub Sponsors.

Thank you for Reading !! See you in the next post.

➡️ Day 25: Protobuf — Proto3, code gen & Spring integration

Resources

📘 Kafka: The Definitive Guide — Chapter 9 (Schema Evolution)
🌐 avro.apache.org/docs/current/spec — Schema Resolution
🌐 docs.confluent.io/platform/current/schema-registry/avro

👉 Link to Medium blog