Gold Schema Compatibility CI Gate (ADR-036 MVP)

[SatoryKono]

Summary

The repository already has a real schema-governance baseline in CI. The remaining gap is an explicit Gold compatibility classifier and gate aligned with ADR-036, so PRs can distinguish compatible Gold contract evolution from breaking Gold contract changes.

Current baseline (2026-04-12)

The repo already contains:

• accepted versioning policy in docs/02-architecture/decisions/ADR-036-gold-contract-versioning-policy.md
• schema-governance workflow in .github/workflows/schema-governance.yml
• blocking checks for:
  • generated schema artifacts freshness
  • contract import validation
  • schema parity / PK coverage
  • representative Silver schema drift
• supporting tooling such as:
  • scripts/schema/generate_schema_artifacts.py
  • scripts/schema/generate_contracts.py
  • src/tools/verify_schema_parity.py
  • src/tools/generate_json_field_typing_inventory.py

What is still missing

What is still not present is a Gold-focused compatibility diff/classification layer that:

• detects Gold contract changes in a PR
• classifies them according to ADR-036 semantics
• produces readable diagnostics
• fails CI only for breaking Gold changes

In other words, schema governance exists, but the ADR-036 Gold compatibility policy is not yet enforced as a first-class CI gate.

Goal

Add an explicit Gold schema compatibility classifier on top of the existing schema-governance workflow without replacing the current gates.

Proposed scope

1. Implement a Gold compatibility rule set derived from ADR-036.

• adding a nullable column = compatible
• removing a column = breaking
• renaming a column = breaking
• narrowing a type = breaking
• changing nullable to non-nullable = breaking
• semantic reinterpretation of an existing field = breaking when detectable through contract metadata or explicit rename/migration markers

2. Add Gold-focused diffing.

• detect Gold contract changes for affected schemas in a PR or compare target/base snapshots
• classify each change using the explicit rule set
• emit human-readable diagnostics explaining why the change is compatible or breaking

3. Integrate the classifier into CI.

• extend .github/workflows/schema-governance.yml or an adjacent workflow step
• keep the existing generated-artifacts / contracts-export / schema-parity / silver-schema-drift checks intact
• fail only when the Gold compatibility classifier identifies breaking changes

4. Document operator/developer workflow.

• explain the rule set
• explain how intentional breaking changes should be introduced under ADR-036
• explain how to update or regenerate any baselines used by the classifier

Concrete evidence anchors

• .github/workflows/schema-governance.yml
• docs/02-architecture/decisions/ADR-036-gold-contract-versioning-policy.md
• docs/D-03 Data Contracts & Schema Specification.md
• docs/reports/great-expectations-spike-2026-04-01.md
• src/tools/verify_schema_parity.py

Out of scope

• downstream consumer registry / notification automation
• replacing Silver schema drift checks with a new framework
• full multi-layer schema governance redesign in this issue
• broad deprecation workflow automation outside Gold contract compatibility

Definition of done

• Gold schema changes are classified explicitly as compatible or breaking
• CI exposes a dedicated Gold compatibility result in addition to the current governance gates
• diagnostics explain the exact Gold diff and classification outcome
• docs explain the rule set, extension path, and intentional-breaking-change workflow

Acceptance tests

• adding a nullable Gold column is classified as compatible
• removing a Gold column is classified as breaking
• renaming a Gold column is classified as breaking
• narrowing a Gold column type is classified as breaking
• changing nullable to non-nullable is classified as breaking
• CI output shows the concrete Gold contract diff in human-readable form
• the existing schema-governance gates remain green/independent when no Gold breaking change is introduced

Related

• ADR-036 Gold contract versioning policy
• existing schema-governance workflow
• complements, and does not replace, parity and Silver drift checks

[SatoryKono]

Retriaged to an ADR-036 compatibility-gate MVP. Registry, notifications, and deprecation workflow automation should follow only after the base CI guardrail exists and proves useful.

[SatoryKono]

Предлагаю re-scope issue под текущую repo reality.

Что уже е���ть:

• workflow Schema Governance уже Ñ���ущеÑ���твует в .github/workflows/schema-governance.yml
• blocking checks на generated artifacts, contract imports и schema parity уже еÑ���Ñ‚ÑŒ
• src/tools/verify_schema_parity.py уже Ñ���равнивает Silver ↔ Gold и иÑ���пользует baseline длÑ��� извеÑ���тных раÑ���хождений
• representative silver schema drift gate уже вÑ���троен в CI

Что пока не выгл���дит закрытым:

• Ñ���внаÑ��� Gold compatibility classification по правилам ADR-036
• readable diff именно в терминах compatible vs breaking
• maintainable baseline именно длÑ��� Gold contract compatibility rules
• policy path длÑ��� intentional approved breaking changes

Предлагаю обновить issue так, чтобы он был не про “���оздать schema governance ��� нул������, а про ���ледующий конкретный шаг.

Suggested updated goal:
Add explicit Gold compatibility classification on top of existing schema-governance gates.

Suggested scope:

• build a Gold-focused compatibility checker on top of existing schema governance
• classify changes using explicit ADR-036 rules
• emit human-readable diagnostics for compatible vs breaking changes
• fail CI only on breaking Gold contract changes

Suggested acceptance criteria:

• adding nullable columns is classified as compatible
• removing/renaming columns is classified as breaking
• narrowing types is classified as breaking
• changing nullable -> non-nullable is classified as breaking
• CI output explains the exact Gold contract diff in human-readable form
• approved extension path is documented for new rules

[SatoryKono]

Предлагаю re-scope issue под текущую repo reality.

Что уже е���ть:

• workflow Schema Governance уже Ñ���ущеÑ���твует в .github/workflows/schema-governance.yml
• blocking checks на generated artifacts, contract imports и schema parity уже еÑ���Ñ‚ÑŒ
• src/tools/verify_schema_parity.py уже Ñ���равнивает Silver ↔ Gold и иÑ���пользует baseline длÑ��� извеÑ���тных раÑ���хождений
• representative silver schema drift gate уже вÑ���троен в CI

Что пока не выгл���дит закрытым:

• Ñ���внаÑ��� Gold compatibility classification по правилам ADR-036
• readable diff именно в терминах compatible vs breaking
• maintainable baseline именно длÑ��� Gold contract compatibility rules
• policy path длÑ��� intentional approved breaking changes

Предлагаю обновить issue так, чтобы он был не про “���оздать schema governance ��� нул������, а про ���ледующий конкретный шаг.

Suggested updated goal:
Add explicit Gold compatibility classification on top of existing schema-governance gates.

Suggested scope:

• build a Gold-focused compatibility checker on top of existing schema governance
• classify changes using explicit ADR-036 rules
• emit human-readable diagnostics for compatible vs breaking changes
• fail CI only on breaking Gold contract changes

Suggested acceptance criteria:

• adding nullable columns is classified as compatible
• removing/renaming columns is classified as breaking
• narrowing types is classified as breaking
• changing nullable -> non-nullable is classified as breaking
• CI output explains the exact Gold contract diff in human-readable form
• approved extension path is documented for new rules

[SatoryKono]

Detailed execution plan proposal for #2516 based on the current repository baseline.

The important framing is that this issue is not about inventing schema governance from zero. The repository already has a working governance baseline in CI: generated-artifact freshness checks, contract import validation, schema parity, and representative Silver schema drift checks. ADR-036 already defines Gold contract versioning policy. The missing piece is a dedicated Gold compatibility classifier/gate that turns ADR-036 rules into an explicit CI verdict.

Phase 1: define the canonical comparison surface.
Before implementing any classifier, decide which artifact is the source of truth for Gold compatibility comparison. That should be the same normalized contract representation already trusted by the current governance flow, not a second parallel schema format. If this is not fixed first, the gate will become noisy or misleading. The output of this phase should be one deterministic compare target for “base vs changed Gold contract.”

Phase 2: implement the minimal ADR-036 rule engine.
Start with a deliberately small rule set that is easy to explain and test:

• adding a nullable Gold column = compatible
• removing a Gold column = breaking
• renaming a Gold column = breaking
• narrowing a Gold type = breaking
• changing nullable -> non-nullable = breaking
  For the MVP, the goal should be deterministic classification of clearly mechanical changes, not over-ambitious semantic inference.

Phase 3: build a normalized Gold diff model.
The classifier should produce a structured diff rather than only a pass/fail verdict. The output should capture which columns were added, removed, renamed, or changed in type/nullability, and it should provide an explanation for the classification. This is necessary both for CI readability and for debugging false positives/false negatives.

Phase 4: integrate the classifier into the existing schema-governance workflow.
Add a dedicated job or step that runs alongside the current schema-governance checks rather than replacing them. The output should clearly distinguish:

• generated artifact freshness
• contract import health
• schema parity / PK coverage
• Silver drift status
• Gold compatibility classification
  This separation matters because Gold compatibility is a governance policy gate, not just another parity check.

Phase 5: define the intentional-breaking-change workflow.
A useful gate must also tell maintainers what to do when a breaking change is intentional. The docs should explain the expected ADR-036 process: version bump, migration notes, dual-service / compatibility window where applicable, and update steps for any baseline or contract artifacts. Without this, the CI gate becomes friction instead of governance.

Phase 6: add deterministic tests.
Add unit tests for the rule engine and at least one integration-style test that compares a baseline Gold contract against a modified version. The tests should prove both sides of the contract: compatible additions pass, and breaking changes fail with readable diagnostics.

Recommended implementation order:

1. choose canonical comparison artifact
2. implement normalized diff model
3. implement ADR-036 classifier rules
4. wire the classifier into CI
5. add diagnostics-focused tests
6. publish remediation / migration guidance

Suggested done criteria for closure:

• Gold contract changes are classified explicitly as compatible or breaking
• CI exposes a dedicated Gold compatibility result
• diagnostics explain the exact Gold diff and why it is allowed or blocked
• maintainers have a documented path for intentional breaking changes under ADR-036
• the existing schema-governance checks remain intact and clearly separated from the new Gold compatibility gate

[SatoryKono]

Status check as of 2026-04-12: this issue is not complete yet and should remain open.

What was verified:

• .github/workflows/schema-governance.yml currently contains generated-artifacts, contracts-export, schema-parity, and representative Silver schema drift checks.
• the repository does have an accepted ADR-036 policy (docs/02-architecture/decisions/ADR-036-gold-contract-versioning-policy.md).
• the repository does not yet appear to have a dedicated Gold compatibility classifier / CI gate implementing ADR-036 semantics.
• docs/reports/great-expectations-spike-2026-04-01.md still refers to #2516 as a future compatibility gate, which is consistent with the code/workflow state.

Conclusion:

• schema governance baseline exists
• ADR-036 policy exists
• the missing enforcement layer still appears to be missing
• therefore this issue should stay open

Practical closeout checklist:

• choose the canonical Gold contract artifact to compare in CI
• implement a normalized Gold diff model
• implement ADR-036 compatibility classification rules
• add a dedicated Gold compatibility step/job to schema governance CI
• add unit/integration tests for both compatible and breaking cases
• document the intentional-breaking-change workflow and CI remediation path

Suggested close condition:
Close only after CI can explicitly classify Gold schema changes as compatible or breaking and block only the breaking cases with readable diagnostics.

[SatoryKono]

Implemented in main at c721d18a39c3178fd2ae2ddb87814b4689968d93.

What is already present:

• dedicated CI execution in .github/workflows/contract-governance-fast-check.yml
• Gold contract diff/classification gate in scripts/ci/validate_schema_classifier_gate.py
• classifier policy/runtime in src/bioetl/domain/services/schema_classifier.py and src/bioetl/domain/types/schema_policy.py
• blocking enforcement for major changes without required version/migration metadata, plus manual_review blocking
• diagnostics artifact emission via contract-schema-classifier-diagnostics.json
• targeted unit coverage in tests/unit/scripts/test_validate_schema_classifier_gate.py

This matches the issue’s acceptance intent: explicit classification, human-readable diagnostics, and CI failure for breaking Gold changes without replacing the existing governance checks.

Closing as completed.