Skip to content

spec(compliance): force_scenario_unsupported — UNKNOWN_SCENARIO on force_* steps grades not_applicable#4228

Draft
bokelley wants to merge 1 commit into3.0.xfrom
claude/issue-4226-force-scenario-unsupported-not-applicable
Draft

spec(compliance): force_scenario_unsupported — UNKNOWN_SCENARIO on force_* steps grades not_applicable#4228
bokelley wants to merge 1 commit into3.0.xfrom
claude/issue-4226-force-scenario-unsupported-not-applicable

Conversation

@bokelley
Copy link
Copy Markdown
Contributor

@bokelley bokelley commented May 8, 2026

Closes #4226

Summary

The compliance runner graded create_media_buy_async as FAILED when a seller returned UNKNOWN_SCENARIO on force_create_media_buy_arm. The storyboard narrative already said this should grade not_applicable, but the runner contract had no machine-readable enforcement layer for explicit force_* phase steps — only fixture_seed_unsupported (auto-injected fixture phase) was documented.

This PR adds the missing spec text so conforming runners can implement the rule.

Changed files:

  • static/compliance/source/universal/storyboard-schema.yaml — adds force_scenario_unsupported block with normative MUST for runners: detect the tuple (comply_test_controller present, resolved payload scenario begins with force_, response {success: false, error: UNKNOWN_SCENARIO}) and grade not_applicable before evaluating declared validations. Documents detection order to prevent misgrading absent-tool cases.
  • static/compliance/source/universal/runner-output-contract.yaml — adds fixture_seed_unsupported, force_scenario_unsupported, and unresolved_scenario_reference as recognized narrower detail values under canonical reason: not_applicable, with the encoding MUST for force_scenario_unsupported.

No storyboard YAML changes — create_media_buy_async.yaml's narrative was already normatively correct.

Non-breaking justification

Patch-eligible per the conformance-harness carve-out and the IETF errata test (playbook lines 261-265): the storyboard's own narrative already required not_applicable grading; runners emitting FAILED were non-conforming against existing normative English. This adds the machine-readable form of a rule already in force — not a new requirement. The canonical reason enum in runner-output-contract.yaml is unchanged; force_scenario_unsupported is a detail sub-value under the existing not_applicable reason.

Sibling-repo fix needed

adcp-client runner needs implementation to honor this contract. File: the UNKNOWN_SCENARIO detection path in the storyboard runner (see src/lib/testing/storyboard/ and DETAILED_SKIP_TO_CANONICAL mapping from PR #611). The detection order MUST place force_scenario_unsupported check after confirming comply_test_controller is present (to avoid misgrading the absent-tool case).

Pre-PR review

  • code-reviewer: approved — no blockers; nit on encoding MUST placement addressed in final diff
  • ad-tech-protocol-expert: approved — non-breaking per spec; nit on resolved-payload wording and detection-order addressed in final diff

Triage-managed PR. This bot does not currently iterate on
review comments or PR conversation threads (only on the source
issue). To unblock:

  • Push fixup commits directly: gh pr checkout <num>
    fix → push.
  • Or re-trigger: comment /triage execute on the source
    issue.

See #3121
for context.

Session: https://claude.ai/code/session_01Uu4K4voWNzFUCUQzKyrsiC


Generated by Claude Code

Sellers that implement comply_test_controller but not a specific force_*
scenario arm return {success: false, error: UNKNOWN_SCENARIO}. The
storyboard narrative already required not_applicable grading; this adds
the machine-readable enforcement layer that was missing.

- storyboard-schema.yaml: adds force_scenario_unsupported block with a
  normative MUST for runners, detection order documentation, and
  distinction from fixture_seed_unsupported / missing_test_controller.
- runner-output-contract.yaml: adds fixture_seed_unsupported,
  force_scenario_unsupported, and unresolved_scenario_reference as
  recognized narrower detail values under reason: not_applicable, with
  the encoding MUST for force_scenario_unsupported.

No storyboard YAML changes — create_media_buy_async.yaml narrative was
already correct. Runner implementation fix tracked in adcp-client.

Closes #4226.

https://claude.ai/code/session_01Uu4K4voWNzFUCUQzKyrsiC
@bokelley bokelley added the claude-triaged Issue has been triaged by the Claude Code triage routine. Remove to re-triage. label May 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

claude-triaged Issue has been triaged by the Claude Code triage routine. Remove to re-triage.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants