60-Day Kafka 4 Learning Plan · Week 4 — Day 26 of 60

---

60-Day Kafka 4 Learning Plan · Week 4 — Schema & Serialization Sources: Kafka: The Definitive Guide Ch.9 · docs.confluent.io/schema-registry/develop/api

Goal

Master the three Schema Registry compatibility modes, understand when _TRANSITIVE variants are needed, apply the right mode for each real-world deployment scenario, and know the risks of changing the mode itself and how to structure it across environments.

1. The mental model — which direction can read which?

BACKWARD ★ (default)

New consumer (v2 schema)  ← reads ←  old message (v1 schema)  ✅
Old consumer (v1 schema)  ← reads ←  new message (v2 schema)  ❌ (not guaranteed)

New schema can read data written by the old schema. Deploy order: consumers first, then producers.

FORWARD

Old consumer (v1 schema)  ← reads ←  new message (v2 schema)  ✅
New consumer (v2 schema)  ← reads ←  old message (v1 schema)  ❌ (not guaranteed)

Old schema can read data written by the new schema. Deploy order: producers first, then consumers.

FULL

New consumer  ← reads ←  old message  ✅
Old consumer  ← reads ←  new message  ✅

Both directions simultaneously. Safest. Most restrictive — only add/remove fields that have default values. Deploy order: any order is safe.

2. _TRANSITIVE variants — checking all history

BACKWARD checks v3 vs v2 only. BACKWARD_TRANSITIVE checks v3 vs v2 and v3 vs v1 — all the way back.

When does _TRANSITIVE actually matter?

Example: v1 → v2 (BACKWARD ok) → v3 (BACKWARD ok vs v2, but BACKWARD fails vs v1).

Without BACKWARD_TRANSITIVE, a consumer reprocessing retained v1 messages with a v3 schema would crash. With transitive, the Registry rejects v3 before it can be registered.

Use _TRANSITIVE whenever:

• Your topic retains messages longer than your deployment cycle
• Consumers can replay messages from the beginning (event sourcing, audit logs)
• You need to guarantee any past consumer version can read any past message

3. Real scenarios with deploy order

Scenario A — Adding a new optional field (BACKWARD)

Schema change: add couponCode with default: null to OrderEvent.

# Set compatibility (already default, but explicit is better)
curl -X PUT http://localhost:8081/config/labs.events-value \
  -d '{"compatibility": "BACKWARD"}'

Deploy order:

1. Deploy updated labs-socket (consumer) with v2 schema

• Now reads old messages: couponCode absent → uses null default ✅

2. Deploy updated labs-api (producer) with v2 schema

• Now writes couponCode field in new messages

Result: Zero downtime, no consumer restart, no lost messages.

Scenario B — Rolling producer upgrade (FORWARD)

Situation: Producer adds a new field; consumers can’t be upgraded yet (legacy system or separate team).

curl -X PUT http://localhost:8081/config/labs.events-value \
  -d '{"compatibility": "FORWARD"}'

Deploy order:

1. Deploy updated labs-api (producer) with v2 schema

• Registry verifies old consumer schema can still read new messages ✅

2. Old consumers silently ignore unknown fields — no crash

3. Consumer team upgrades at their own pace later

Result: Producer and consumer teams deploy independently. No coordination required.

Scenario C — Independent microservices (FULL)

Situation: labs-api and labs-socket deploy on independent schedules with no coordination.

curl -X PUT http://localhost:8081/config/labs.events-value \
  -d '{"compatibility": "FULL"}'

Constraint: Developers may only add or remove fields that carry default values.

Deploy order: Any order is safe — Registry enforces compatibility in both directions.

Result: Either service can deploy first. Both read each other’s data safely. Breaking schemas are caught at CI time, not at runtime.

4. Verify compatibility before registering

Pre-registration check (use in CI/CD pipelines)

curl -X POST \
  http://localhost:8081/compatibility/subjects/labs.events-value/versions/latest \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"schema": "{ ...new schema json... }"}'

# Compatible response
{"is_compatible": true}

# Incompatible response
{
  "is_compatible": false,
  "messages": ["Incompatibility{type:READER_FIELD_MISSING_DEFAULT_VALUE, ...}"]
}

Check against all versions (for TRANSITIVE safety)

# Check against every registered version, not just latest
curl -X POST \
  http://localhost:8081/compatibility/subjects/labs.events-value/versions \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"schema": "{ ...new schema json... }"}'

Maven plugin approach (Day 27 preview)

<plugin>
  <groupId>io.confluent</groupId>
  <artifactId>kafka-schema-registry-maven-plugin</artifactId>
  <version>7.6.1</version>
  <configuration>
    <schemaRegistryUrls>
      <param>http://localhost:8081</param>
    </schemaRegistryUrls>
    <subjects>
      <labs.events-value>src/main/avro/order-event.avsc</labs.events-value>
    </subjects>
  </configuration>
  <executions>
    <execution>
      <phase>verify</phase>
      <goals><goal>test-compatibility</goal></goals>
    </execution>
  </executions>
</plugin>

Run mvn verify to fail the build if a schema change is incompatible.

5. Changing the compatibility mode itself is a risky operation

Everything above assumes the compatibility mode for a subject is set once and stays stable. Changing the mode itself doesn’t retroactively validate anything — it only affects checks performed on future registrations from that point forward.

Subject history: v1, v2, v3 all registered under BACKWARD (each valid against its predecessor)

Someone changes the subject's config to FULL

Registry does NOT go back and re-validate v1→v2→v3 under FULL —
those registrations already happened and are grandfathered in

Next registration attempt (v4) IS checked under the new FULL rules

The real risk: loosening a mode (e.g. FULL → BACKWARD) can silently reopen the door to changes that were previously blocked, and the team may not remember why the stricter mode was chosen in the first place. Tightening a mode (e.g. BACKWARD → FULL_TRANSITIVE) can suddenly reject a schema change that would have been fine under the old rules — surprising a developer who didn’t realize the subject’s config had changed.

Practical guidance: treat a compatibility mode change with the same review rigor as the schema change itself — it’s a policy decision about the topic’s future evolution constraints, not a routine config tweak. Document why a subject deviates from your org’s default mode (§10’s config drift monitoring helps catch undocumented deviations).

6. Environment-specific compatibility strategy

Different environments legitimately warrant different compatibility rigor — but this needs to be a deliberate, documented strategy, not an accident of “we never got around to setting it in staging.”

# Set org-wide default once, override per-subject only when a topic's
# retention/consumer characteristics genuinely warrant it
curl -X PUT http://localhost:8081/config -d '{"compatibility": "BACKWARD"}'

curl -X PUT http://localhost:8081/config/audit.events-value \
  -d '{"compatibility": "FULL_TRANSITIVE"}'   # this specific topic has different needs

Anti-pattern to avoid: having staging run NONE “because it’s faster” while production runs BACKWARD. This means the first time a genuinely incompatible schema is caught is in production — exactly backwards from what staging is for.

7. Compatibility groups — versioning by application release, not just schema shape

Confluent’s compatibility groups feature (via the confluent:version metadata property or Schema Registry’s group configuration) lets you scope compatibility checks to schemas sharing a group identifier — useful when multiple schema versions need to coexist intentionally across major application releases, rather than treating every registration as part of one single linear history.

{
  "schema": "{ ... }",
  "metadata": {
    "properties": {
      "confluent:version": "2.0"
    }
  }
}

Why this matters beyond the basic modes: the standard BACKWARD/FORWARD/FULL modes assume a single, continuously-evolving schema lineage. A compatibility group lets a major version boundary (e.g. “v2.0 of the event contract”) reset what “latest” means for compatibility purposes — useful for a deliberate breaking change tied to a coordinated major release, rather than an accidental incompatible schema slipping through.

Use sparingly. This is an escape hatch for genuinely coordinated major-version transitions (with a migration plan, not just “we wanted to break the schema”), not a way to bypass compatibility checking for convenience. Most teams will never need this — the standard modes plus the migration pattern from Day 22 §10 (parallel topic) cover the vast majority of real breaking-change needs.

8. Monitoring for compatibility mode drift

Since mode changes (§5) are policy decisions, not just config, they should be observable and auditable — not something that silently changed six months ago and nobody remembers why.

# Audit script: compare current subject configs against the org's expected baseline
for subject in $(curl -s http://localhost:8081/subjects | jq -r '.[]'); do
  current=$(curl -s "http://localhost:8081/config/$subject" | jq -r '.compatibilityLevel // "GLOBAL_DEFAULT"')
  echo "$subject: $current"
done

Practical habit: run this kind of audit as a periodic CI job (or a scheduled script) and diff against a checked-in expected-config file — treat unexpected compatibility mode drift the same way you’d treat unexpected infrastructure drift, since it has the same “invisible until it causes an incident” failure profile as, say, an accidentally-changed min.insync.replicas (Day 10 §11’s monitoring philosophy applies equally here).

9. Testing all three scenarios in CI

A CI schema-validation step should assert the actual deploy-order constraint the mode implies, not just “the compatibility endpoint returned true” — that only tells you the schema change is allowed, not that your team’s actual deploy plan matches what the mode requires.

@Test
void backwardCompatibilityMatchesPlannedDeployOrder() {
    // Given the subject is configured BACKWARD, assert the PLAN is "consumers deploy first"
    // — a test that only checks Registry compatibility but ships a producer-first
    // deployment script for a BACKWARD-only topic is a process gap the schema check can't catch
    assertThat(deploymentPlan.getDeployOrder("labs.events"))
        .isEqualTo(DeployOrder.CONSUMERS_FIRST);
}

The gap this catches: it’s entirely possible to pass every Registry compatibility check and still get bitten, if the actual CI/CD pipeline or runbook deploys producers and consumers in the wrong order for the configured mode. The schema check and the deployment orchestration need to agree — testing them together (or at least documenting the coupling clearly) closes a real operational gap.

10. Common pitfalls

• Treating a compatibility mode change as a routine config tweak — it’s a policy decision with real consequences for what future schema changes are allowed or rejected (§5)
• Running looser compatibility rules in staging than production — defeats the purpose of staging as a pre-production safety net (§6)
• Forgetting non-transitive modes only check the immediately preceding version — this is the same Day 24 §6 gap, worth re-emphasizing here since it’s the most common source of “but the compatibility check passed!” surprises
• Reaching for compatibility groups (§7) to bypass an inconvenient compatibility failure — it’s designed for coordinated major-version transitions, not a shortcut around legitimate compatibility enforcement
• No visibility into compatibility mode drift over time — a subject’s mode silently changing (or being set inconsistently across environments) is exactly the kind of “invisible until an incident” risk worth actively auditing (§8)

Key Takeaways

• BACKWARD: new reads old → upgrade consumers first (default, most common)
• FORWARD: old reads new → upgrade producers first (legacy consumer scenario)
• FULL: both directions → any deploy order, safest for independent microservices
• _TRANSITIVE variants check against all historical versions, not just the latest
• Changing a subject’s compatibility mode is a policy decision, not a routine tweak — it doesn’t retroactively re-validate prior registrations and can silently loosen or tighten future ones
• Staging should run the same compatibility mode as production — running it looser defeats staging’s purpose
• Compatibility groups exist for deliberate, coordinated major-version transitions — not a bypass for inconvenient failures
• Audit compatibility mode configuration periodically — drift is invisible until it causes an incident
• Use the /compatibility REST endpoint in CI to catch breaking schemas before merge
• NONE is for dev only — never use in production topics

---

Support me through GitHub Sponsors.

Thank you for Reading !! See you in the next post.

Next

➡️ Day 27: Schema CI/CD — validate schemas in pipeline gates

Resources

• 📘 Kafka: The Definitive Guide — Chapter 9 (Schema Evolution)
• 🌐 docs.confluent.io — Schema Registry API
• 🌐 docs.confluent.io — Schema Compatibility
• 🌐 docs.confluent.io — Schema Metadata and Compatibility Groups

👉 Link to Medium blog