60-Day Kafka 4 Learning Plan · Week 4 — Day 27 of 60

60-Day Kafka 4 Learning Plan · Week 4 — Schema & Serialization Sources: Kafka: The Definitive Guide Ch.9 · docs.confluent.io — Maven Plugin (https://docs.confluent.io/platform/current/schema-registry/develop/maven-plugin.html)

Goal

Build a schema validation pipeline that catches breaking schema changes on every pull request, automatically registers schemas in dev on merge, promotes schemas through staging to production with escalating compatibility gates, and handles the realities of concurrent merges, bad registrations, and credential security.

1. Why schema CI/CD?

Without gates: a developer merges a breaking schema change → labs-socket consumers crash in production reading new messages they can’t deserialize.

With gates: the PR build fails at compatibility check → breaking change caught before it ever reaches main.

PR opens → schema lint → compat check → merge → register in dev → promote to prod
              ❌ fail=safe   ❌ fail=safe    ✅ auto on merge    ✅ on release tag

The two fail-fast gates are the most important: everything downstream is only as safe as what passes through them.

2. Confluent Maven plugin — test-compatibility goal

<!-- pom.xml -->
<plugin>
  <groupId>io.confluent</groupId>
  <artifactId>kafka-schema-registry-maven-plugin</artifactId>
  <version>7.6.1</version>
  <configuration>
    <schemaRegistryUrls>
      <param>${schema.registry.url}</param>
    </schemaRegistryUrls>
    <subjects>
      <!-- map subject name → local schema file -->
      <labs.events-value>src/main/avro/order-event.avsc</labs.events-value>
      <labs.notifications-value>src/main/avro/notification.avsc</labs.notifications-value>
    </subjects>
  </configuration>
  <executions>
    <execution>
      <phase>verify</phase>
      <goals>
        <!-- test-compatibility: check only, does NOT register -->
        <goal>test-compatibility</goal>
      </goals>
    </execution>
  </executions>
</plugin>

# Run locally or in CI — fails with non-zero exit if incompatible
mvn verify -Dschema.registry.url=http://localhost:8081

# Output on failure:
# [ERROR] Schema labs.events-value is not compatible with latest version
# [ERROR] Incompatibility{type:READER_FIELD_MISSING_DEFAULT_VALUE, ...}

3. GitHub Actions — schema validation workflow

# .github/workflows/schema-check.yml
name: Schema Compatibility Check

on:
  pull_request:
    # Only trigger when schema files change — avoids running on every PR
    paths:
      - 'src/main/avro/**'
      - 'src/main/proto/**'

jobs:
  schema-check:
    runs-on: ubuntu-latest

    services:
      # Spin up a Schema Registry instance for the compatibility check
      kafka:
        image: apache/kafka:4.0.0
        ports: ['9092:9092']
      schema-registry:
        image: confluentinc/cp-schema-registry:7.6.1
        ports: ['8081:8081']
        env:
          SCHEMA_REGISTRY_HOST_NAME: schema-registry
          SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: kafka:9092
          SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081

    steps:
      - uses: actions/checkout@v4

      - name: Set up Java 21
        uses: actions/setup-java@v4
        with:
          java-version: '21'
          distribution: 'temurin'
          cache: maven

      - name: Wait for Schema Registry
        run: |
          for i in $(seq 1 30); do
            curl -sf http://localhost:8081/subjects && break || sleep 2
          done

      - name: Check schema compatibility
        run: mvn verify -Dschema.registry.url=http://localhost:8081 --no-transfer-progress

Note the pattern here: this job spins up an ephemeral Registry just for the compatibility check, seeded with nothing. That verifies the schema is internally well-formed, but it’s checking compatibility against an empty history, not your real dev/staging Registry’s actual current schema. See §7 for why the real check that matters still happens against the live Registry on merge, not in this PR job alone.

4. Environment promotion strategy

DEV — automatic on merge to main

# Mode: BACKWARD (default)
# auto.register.schemas: true in application.yml
# No manual step needed — schemas are registered by the producer on first publish

STAGING — gated on release branch

# .github/workflows/schema-register.yml
name: Register Schemas in Staging

on:
  push:
    branches: ['release/**']
    paths: ['src/main/avro/**']

jobs:
  register-staging:
    runs-on: ubuntu-latest
    environment: staging          # requires environment approval in GitHub
    steps:
      - uses: actions/checkout@v4
      - name: Set compatibility to BACKWARD_TRANSITIVE
        run: |
          curl -X PUT ${{ secrets.STAGING_SCHEMA_REGISTRY_URL }}/config \
            -H "Content-Type: application/vnd.schemaregistry.v1+json" \
            -d '{"compatibility": "BACKWARD_TRANSITIVE"}'
      - name: Register schemas in staging
        run: mvn schema-registry:register
          -Dschema.registry.url=${{ secrets.STAGING_SCHEMA_REGISTRY_URL }}

PROD — manual gate on release tag

on:
  push:
    tags: ['v*']            # trigger on version tags only

jobs:
  register-prod:
    environment: production   # requires manual approval from prod gatekeepers
    steps:
      - name: Verify FULL_TRANSITIVE compatibility
        run: mvn verify
          -Dschema.registry.url=${{ secrets.PROD_SCHEMA_REGISTRY_URL }}
          # pom profile switches to FULL_TRANSITIVE for prod
      - name: Register schemas in production
        run: mvn schema-registry:register
          -Dschema.registry.url=${{ secrets.PROD_SCHEMA_REGISTRY_URL }}

5. Register schemas on merge

# .github/workflows/schema-register.yml  (dev auto-register)
name: Register Schemas in Dev

on:
  push:
    branches: [main]
    paths: ['src/main/avro/**']

jobs:
  register-dev:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Register schemas in dev Registry
        run: |
          mvn schema-registry:register \
            -Dschema.registry.url=${{ secrets.DEV_SCHEMA_REGISTRY_URL }}

Two separate goals to remember:

test-compatibility — checks only, does not register. Use in PR builds.
schema-registry:register — registers the schema. Use on merge/deploy.

6. Making schema changes reviewable — a human-readable PR diff

A raw .avsc/.proto diff in a PR is hard for a reviewer to reason about at a glance — ”is this field addition safe?” requires mentally re-deriving the compatibility rules from Day 22/24/25. Posting the actual compatibility check result as a PR comment turns that into a fact instead of a guess.

- name: Check schema compatibility and comment on PR
  run: |
    result=$(curl -s -X POST \
      "${{ secrets.DEV_SCHEMA_REGISTRY_URL }}/compatibility/subjects/labs.events-value/versions/latest" \
      -H "Content-Type: application/vnd.schemaregistry.v1+json" \
      -d "{\"schema\": $(jq -Rs . < src/main/avro/order-event.avsc)}")
    echo "COMPAT_RESULT=$result" >> "$GITHUB_ENV"

- name: Post compatibility result as PR comment
  uses: actions/github-script@v7
  with:
    script: |
      github.rest.issues.createComment({
        issue_number: context.issue.number,
        owner: context.repo.owner,
        repo: context.repo.repo,
        body: `Schema compatibility check: \`${process.env.COMPAT_RESULT}\``
      });

Why this is worth the extra step: it turns “trust the CI gate turned green” into “here’s the specific compatibility verdict and the incompatibility messages if any” directly in the review thread — closing the gap between the pipeline caught it and the reviewer actually understood what was being approved.

7. What happens when a bad schema slips through anyway

Despite the gates, something can still get registered that shouldn’t have been — a race condition (§8), a Registry misconfigured to NONE in an environment it shouldn’t be, or a manual curl bypassing CI entirely. Schema Registry’s recovery options here are limited, which is exactly why prevention (the gates above) matters more than remediation.

You cannot simply “roll back” to a previous schema version the way you’d revert a code commit — old versions remain registered and their IDs stay valid (any message already written referencing that schema ID must remain readable), but you can’t retroactively un-register a bad version without breaking any consumer that might resolve it.
The real fix is forward, not backward: register a new corrected version that’s compatible with what’s already there, following the same process as any other schema change. Treat a bad registration like a bad database migration — you write a follow-up migration, you don’t edit history.
Soft-deleting a subject/version (DELETE /subjects/{subject}/versions/{version}) is possible but only safe if you’re certain no message ever referenced that exact schema ID — in practice, this is hard to guarantee on a live topic, and a hard delete additionally frees the version number for potential reuse issues. Confluent’s docs explicitly caution against deleting versions in production for this reason.

This is why the gates matter more than the cleanup story. There’s no clean undo button once a bad schema is live and consumers may have already fetched it — the entire design of this pipeline (§1–§6) is about not needing one.

8. Concurrent PRs — a race condition worth knowing about

Two PRs modifying the same subject’s schema, both passing their individual test-compatibility checks against the Registry state at the time each ran, can still conflict if merged in quick succession — PR A’s compatibility check ran against v3, PR B’s also ran against v3 (before A’s merge registered v4), and after both merge, whichever registers second is actually being checked against a state its own CI run never saw.

PR A: checks against v3 → passes → merges → registers v4
PR B: checks against v3 (ran before A merged) → passes → merges → registers v5
                                                                    ↑ this check was
                                                                      against STALE v3,
                                                                      not the real v4

Mitigation: make the merge-time registration step (§5) the authoritative check, not just the PR-time one — if schema-registry:register fails post-merge because the schema is incompatible with what actually landed first, that failure needs to page someone or block the deploy, not be silently ignored. Treat merge-queue serialization (GitHub’s merge queue feature, or a simple “one schema PR in flight at a time” convention for high-change-frequency subjects) as a real mitigation, not overkill, for subjects with frequent concurrent schema changes.

9. Securing Registry credentials in CI

The workflows above reference secrets.DEV_SCHEMA_REGISTRY_URL etc. — worth being explicit about what needs protecting beyond just “it’s a secret”:

Production Registry credentials should never be reachable from a PR-triggered job (§3’s job) — only from jobs gated behind environment: production with required approvals, exactly as shown in §4. A malicious or compromised PR (e.g. from a fork) must not be able to reach production credentials at all.
Use GitHub’s environment protection rules, not just secret scoping — a secret referenced in a workflow is still accessible to whatever triggers that job; the environment: gate is what actually enforces the approval step, not just where the secret happens to be stored.
Rotate Schema Registry credentials on the same cadence as any other production credential — they’re a real production access path (can register/delete schemas, per Day 23 §9) and should be treated with equivalent operational rigor, not as a lower-stakes CI convenience secret.

10. Monitoring the pipeline itself

Treat a merge-time registration failure as a P1, not a retry-and-move-on event — it means the PR-time safety check and reality diverged, which is exactly the gap this whole pipeline exists to close.

11. Common pitfalls

Trusting the PR-time compatibility check as the final word — it only reflects the Registry state at the time it ran; the merge-time registration (§5, §8) is the actual authoritative check
Storing production Registry credentials without environment: protection rules — secret scoping alone doesn’t enforce an approval gate (§9)
Assuming a bad registration can be cleanly rolled back — plan for forward-fix, not undo (§7)
Triggering the compatibility-check workflow on every PR instead of scoping to schema file paths — wastes CI time and slows feedback for unrelated changes
Not treating merge-time registration failures as urgent — a failure there means the safety net actually caught something the PR-time check missed, not routine noise

Key Takeaways

Schema compatibility must be gated in CI — not caught at runtime in production
kafka-schema-registry-maven-plugin: test-compatibility on PR, register on merge
GitHub Actions: trigger on avro/** or proto/** path changes only — fast, focused feedback
Promote schemas through environments: dev (auto) → staging (gated) → prod (manual)
Post the compatibility check result as a PR comment — turns “CI is green” into an actual reviewable fact
There’s no clean rollback for a bad schema registration — treat it like a database migration and fix forward
Concurrent PRs touching the same subject can race past their individual PR-time checks — the merge-time registration is the real authoritative gate
Production Registry credentials need environment: approval gates, not just secret scoping
Store Registry URLs as GitHub Secrets — never hardcode env-specific URLs in workflow files
Escalate mode per env: BACKWARD → BACKWARD_TRANSITIVE → FULL_TRANSITIVE

Support me through GitHub Sponsors.

Thank you for Reading !! See you in the next post.

➡️ Day 28: Week 4 lab — migrate JSON topic to Avro safely

Resources

📘 Kafka: The Definitive Guide — Chapter 9 (Administering Kafka)
🌐 docs.confluent.io — Schema Registry Maven Plugin
🌐 docs.github.com — Environments & Approvals
🌐 docs.github.com — Merge queue

👉 Link to Medium blog