diff --git a/.changeset/fix-force-scenario-unsupported-not-applicable.md b/.changeset/fix-force-scenario-unsupported-not-applicable.md new file mode 100644 index 0000000000..12bf6c0920 --- /dev/null +++ b/.changeset/fix-force-scenario-unsupported-not-applicable.md @@ -0,0 +1,17 @@ +--- +"adcontextprotocol": patch +--- + +spec(compliance): document `force_scenario_unsupported` — UNKNOWN_SCENARIO on force_* controller steps grades not_applicable + +Sellers that implement `comply_test_controller` but have not implemented a specific `force_*` scenario arm (e.g., `force_create_media_buy_arm`) correctly return `{success: false, error: UNKNOWN_SCENARIO}`. The storyboard narrative in `create_media_buy_async.yaml` already said this grades `not_applicable` — that narrative was normative English. The runner contract, however, had no machine-readable enforcement layer for force_* scenarios (only `fixture_seed_unsupported` for auto-injected seed phases), so conforming runners were implementing FAILED instead of not_applicable. + +Patch-eligibility justification (IETF errata test, playbook lines 261-265): the storyboard's own normative narrative text already required not_applicable; any runner grading FAILED was non-conforming against that existing MUST. This change adds the machine-readable form of a rule that was already in force. A conformant 3.0.0 implementation of the surrounding behavior would already have honored the narrative — the schema text closing the gap is an errata clarification, not a new requirement. + +Changes: +- `storyboard-schema.yaml`: adds `force_scenario_unsupported` alongside `fixture_seed_unsupported`, with a normative MUST: detect the tuple (comply_test_controller IS present, resolved payload scenario begins with `force_`, response {success: false, error: UNKNOWN_SCENARIO}) and grade not_applicable before evaluating declared validations. Documents detection order to prevent misgrading absent-tool cases. +- `runner-output-contract.yaml`: adds `fixture_seed_unsupported`, `force_scenario_unsupported`, and `unresolved_scenario_reference` as recognized narrower detail values under canonical reason `not_applicable`, with the encoding MUST for `force_scenario_unsupported`. + +No storyboard YAML changes — `create_media_buy_async.yaml`'s narrative was already correct; this closes the machine-readable gap the runner was missing. Runner implementation fix tracked in adcp-client (sibling-repo). + +Closes #4226. diff --git a/static/compliance/source/universal/runner-output-contract.yaml b/static/compliance/source/universal/runner-output-contract.yaml index b2fb544659..d00da84795 100644 --- a/static/compliance/source/universal/runner-output-contract.yaml +++ b/static/compliance/source/universal/runner-output-contract.yaml @@ -322,6 +322,34 @@ skip_result: DETAILED_SKIP_TO_CANONICAL in the conforming implementation for a concrete mapping. + Recognized narrower detail values for reason: not_applicable (MUST use + these exact strings when the cause matches, so dashboards can bucket them): + + fixture_seed_unsupported — a seed_* controller scenario in the + auto-injected fixtures phase returned + UNKNOWN_SCENARIO (seller declared + comply_test_controller but has not + implemented that seed scenario). + + force_scenario_unsupported — a force_* controller scenario in an + explicit storyboard phase returned + UNKNOWN_SCENARIO. Runners MUST detect + the tuple (comply_test_controller + present, resolved payload scenario + begins with `force_`, response + {success: false, error: UNKNOWN_SCENARIO}) + and grade not_applicable before applying + the step's declared validations; encode + `force_scenario_unsupported` in + skip_result.detail. See + storyboard-schema.yaml > + force_scenario_unsupported. + + unresolved_scenario_reference — a requires_scenarios reference did not + resolve against the source tree (authoring + bug; distinct from the two above, which + are agent-coverage gaps). + run_summary: description: | Runners MUST expose a top-level summary for every run. UI surfaces diff --git a/static/compliance/source/universal/storyboard-schema.yaml b/static/compliance/source/universal/storyboard-schema.yaml index 65456f8c03..5c2695e762 100644 --- a/static/compliance/source/universal/storyboard-schema.yaml +++ b/static/compliance/source/universal/storyboard-schema.yaml @@ -852,6 +852,35 @@ # UNKNOWN_SCENARIO). The storyboard grades not_applicable (coverage gap), # not failed. # +# force_scenario_unsupported +# Emitted when a `comply_test_controller` step in an explicit storyboard +# phase (not the auto-injected fixture phase) receives +# {success: false, error: UNKNOWN_SCENARIO} for a scenario whose name +# begins with `force_`. This is the explicit-phase analogue of +# `fixture_seed_unsupported`: the seller advertises comply_test_controller +# but has not implemented the specific force_* scenario arm the storyboard +# requires. The storyboard grades not_applicable (coverage gap), not +# failed. +# +# Runners MUST detect this tuple — comply_test_controller IS advertised +# (missing_test_controller check already passed), the resolved wire +# payload's `scenario` field begins with `force_`, and the response +# carries {success: false, error: UNKNOWN_SCENARIO} — and apply +# not_applicable grading before evaluating the step's declared +# validations. The narrower reason MUST be encoded in skip_result.detail +# per the convention in runner-output-contract.yaml; the canonical reason +# remains not_applicable. +# +# Detection order: `missing_test_controller` (tool absent) fires first; +# `force_scenario_unsupported` fires only when the tool IS present. A +# runner that checks `force_scenario_unsupported` before confirming tool +# presence would misgrade absent-tool cases whose scenario name starts +# with `force_`. +# +# Distinct from `fixture_seed_unsupported` (auto-injected fixture phase, +# seed_* scenarios) and from `missing_test_controller` (the agent did not +# advertise comply_test_controller at all). Closes #4226. +# # unresolved_scenario_reference # A detailed sub-reason under the canonical skip `reason: # not_applicable`. Emitted when a parent storyboard's