Provider Contract Drift Checks MVP for External Schemas and Fields

[SatoryKono]

Summary

Split out the provider contract-drift track from #2511 into a small executable MVP for detecting high-signal external schema or field drift.

Why

The parent roadmap identifies contract drift as a real risk area, but it is too broad to execute as a roadmap item alone. This issue narrows the work to a minimal, high-value CI check.

Scope

• Identify one or a small set of providers where schema or field drift risk is highest.
• Define a lightweight baseline for expected fields or payload shape.
• Add focused checks that flag drift with readable diagnostics.
• Keep the implementation small enough to extend incrementally provider by provider.

Out of Scope

• Full platform-wide contract testing framework
• Broad compatibility governance across every layer
• Full consumer notification workflows
• Gold-layer compatibility checks already covered separately by Gold Schema Compatibility CI Gate (ADR-036 MVP) #2516

Acceptance Criteria

• At least one high-risk provider is covered by a focused drift check.
• Drift produces readable diagnostics in test or CI output.
• The baseline and extension path are documented.
• The implementation aligns with the current testing and governance model.

Parent

• Split out of Testing Roadmap: unit standards, VCR integration, contract drift, data validation gates #2511

Related

• Complements Gold Schema Compatibility CI Gate (ADR-036 MVP) #2516 rather than replacing it

[SatoryKono]

Checked the repository seams before proposing an implementation path for #2599. The important conclusion is that this should not be a brand-new framework rollout: the repo already has a bounded provider snapshot drift mechanism, so the best MVP is to complete and extend that existing slice.

What already exists

The repo already has a concrete snapshot-based provider drift layer:

• helper for provider payload snapshot checks:
  • tests/contract/_provider_contract_drift.py
• local registry validation:
  • tests/contract/test_provider_contract_snapshot_registry.py
• governance wiring in the matrix:
  • configs/quality/test_matrix.yaml
• snapshot registry docs:
  • tests/fixtures/contracts/README.md
• current bounded rollout slice:
  • crossref
  • openalex
  • semanticscholar

Those providers already use live contract suites with assert_provider_probe_matches_snapshot(...) against canonical fixture snapshots.

Recommended MVP scope

I recommend using PubMed as the next provider for #2599.

Why PubMed:

• there is already a live provider contract suite:
  • tests/contract/test_pubmed_contract.py
• it already exercises JSON endpoints that fit the current helper model
• it stays within the publication/provider slice instead of introducing a new family
• it demonstrates that the current bounded snapshot system extends provider-by-provider rather than remaining a pilot-only mechanism

I would explicitly keep XML-heavy endpoints like efetch out of the MVP and focus only on JSON payload probes.

Proposed implementation plan

1. Treat the current bounded snapshot system as the official MVP base

Do not introduce a parallel framework.

Re-use:

• tests/contract/_provider_contract_drift.py
• tests/contract/test_provider_contract_snapshot_registry.py
• configs/quality/test_matrix.yaml
• tests/fixtures/contracts/README.md

2. Add PubMed to the contract snapshot registry

• create tests/fixtures/contracts/pubmed/v1.json
• add pubmed under fixture_governance.contract_snapshot_registry.providers in configs/quality/test_matrix.yaml
• update any exact provider-set assertions if needed

3. Add PubMed snapshot assertions to the live contract suite

Extend tests/contract/test_pubmed_contract.py with snapshot-backed probes for JSON endpoints only.

Recommended probes:

• esearch_endpoint
• esummary_endpoint
• einfo_pubmed_details

For each probe, add assert_provider_probe_matches_snapshot(...).

4. Keep the baseline intentionally narrow

The MVP should protect a small provider-facing payload-shape surface, not entire upstream responses.

Use the existing path-type model from tests/contract/_provider_contract_drift.py and freeze only a handful of high-signal paths.

Do not snapshot whole payloads.

5. Document the extension path

Update tests/fixtures/contracts/README.md so the next provider addition is obvious.

Document:

• how to choose probes
• how to add a new provider snapshot file
• when to use UPDATE_SNAPSHOTS=1
• what these snapshots protect vs. what they intentionally do not protect

6. Keep governance aligned with the current testing model

Use the existing matrix/architecture guards rather than adding a second governance layer.

The implementation should continue to work with:

• tests/contract/test_provider_contract_snapshot_registry.py
• tests/architecture/test_test_matrix_coverage.py

Expected deliverables

• tests/fixtures/contracts/pubmed/v1.json
• updated configs/quality/test_matrix.yaml
• snapshot assertions in tests/contract/test_pubmed_contract.py
• updated tests/fixtures/contracts/README.md
• any small registry/architecture guard adjustments required for the new provider

Verification plan

• run registry shape tests
• run architecture matrix coverage tests
• run targeted pubmed contract tests
• run once with UPDATE_SNAPSHOTS=1 to generate the baseline
• run again without UPDATE_SNAPSHOTS to verify normal drift-check behavior

Why this is the right-sized MVP

This keeps #2599 small and executable:

• extends an existing working mechanism
• adds one high-risk provider with readable diagnostics
• documents the baseline and extension path
• aligns with the current testing/governance model instead of competing with it

That matches the issue’s acceptance criteria much better than trying to design a broader provider-contract platform in one step.

[SatoryKono]

Implemented the bounded MVP slice proposed for #2599 by extending the existing provider snapshot drift mechanism rather than introducing a new framework.

What changed:

• added pubmed to the bounded provider snapshot registry in configs/quality/test_matrix.yaml
• added canonical snapshot baseline file:
  • tests/fixtures/contracts/pubmed/v1.json
• extended the local registry test to include pubmed:
  • tests/contract/test_provider_contract_snapshot_registry.py
• extended the live PubMed contract suite with snapshot-backed JSON probes:
  • esearch_endpoint
  • esummary_endpoint
  • einfo_database_list
  • implemented in tests/contract/test_pubmed_contract.py
• updated snapshot registry docs in tests/fixtures/contracts/README.md
• updated architecture governance so the bounded managed slice now includes pubmed:
  • tests/architecture/test_test_matrix_coverage.py

One intentional implementation choice: for the MVP I used einfo_database_list rather than einfo_pubmed_details because the current snapshot helper freezes narrow path/type expectations and the existing PubMed test already documents that dbinfo may vary between list and dict. Keeping the MVP on stable JSON probes reduces noise and keeps the extension path incremental.

Focused verification completed:

• tests/contract/test_provider_contract_snapshot_registry.py ✅
• tests/architecture/test_test_matrix_coverage.py -k bounded_contract_snapshot_registry_matches_managed_slice ✅
• pytest --collect-only tests/contract/test_pubmed_contract.py ✅

Notes:

• I did not run the live PubMed network tests in this pass, so this is governance-backed and syntax/collection-backed verification rather than a live API confirmation run.
• The implementation stays aligned with the current testing/governance model and extends the existing bounded publication snapshot slice provider-by-provider, which is exactly the intended MVP shape for this issue.

[SatoryKono]

Closing audit for #2599.

Implemented a bounded Provider Contract Drift Checks MVP by extending the existing snapshot-backed contract drift mechanism to pubmed rather than introducing a new framework.

Completed work:

• Added pubmed to the contract snapshot registry in configs/quality/test_matrix.yaml
• Added the pubmed baseline snapshot at tests/fixtures/contracts/pubmed/v1.json
• Extended tests/contract/test_pubmed_contract.py with snapshot-backed probes for:
  • esearch_endpoint
  • esummary_endpoint
  • einfo_database_list
• Updated governance/docs coverage in:
  • tests/contract/test_provider_contract_snapshot_registry.py
  • tests/architecture/test_test_matrix_coverage.py
  • tests/fixtures/contracts/README.md

Intentional scope decision:

• Kept the MVP on stable JSON probes only
• Did not include XML-heavy efetch
• Did not snapshot einfo_pubmed_details, because the current helper is better suited to bounded stable path/type checks than to that less stable response shape

Verification completed:

• registry tests passed
• architecture governance check for managed snapshot slice passed
• tests/contract/test_pubmed_contract.py collected successfully
• live network run passed with:
  • BIOETL_LIVE_API_TESTS=true
  • BIOETL_NETWORK_TESTS=true

This satisfies the MVP acceptance for #2599: at least one additional high-signal provider is covered, diagnostics remain readable through the existing snapshot helper, and the baseline/extension path is documented.