From 5e1228d6c4276f3e4d59389c4c5fc21ee681e15e Mon Sep 17 00:00:00 2001 From: Brian O'Kelley Date: Mon, 11 May 2026 06:10:09 -0400 Subject: [PATCH] docs(verification): storyboards vs. scenarios disambiguation (closes #4035) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three things in AdCP share the word "scenarios" and they aren't the same. External implementers grepping `node_modules/@adcp/sdk/dist/lib/testing/ scenarios/media-buy.js` for `buying_mode` and concluding sales-guaranteed coverage is missing the proposal/refine/finalize lifecycle have a reasonable evidence path — but the conclusion is wrong, because the YAML storyboards do cover that lifecycle (`proposal_finalize`, `refine_ products`). The scenario file just predates `comply()` and is no longer the source of truth. New page `docs/building/verification/storyboards-vs-scenarios.mdx` names the three things explicitly: 1. **Storyboards** (YAML at `/compliance/{version}/...`, run via `comply()`) — the conformance specification. Normative. 2. **`comply_test_controller` scenarios** (`force_*`, `simulate_*`, `seed_*` operations the seller exposes) — protocol-level tool operations that storyboards use to drive deterministic state. Normative, but different from storyboards. 3. **`@adcp/sdk/testing/scenarios/*.ts`** — legacy TypeScript test runners predating storyboard-driven `comply()`. Non-normative. May differ from storyboard contents. Plus a three-line decoder: "if you see a `.yaml` under `/compliance/...`, trust as spec; if you see a `.ts` under `testing/scenarios/`, don't." Cross-linked from `conformance.mdx` and `validate-your-agent.mdx` via `` callouts pointing readers at the disambiguation page when they might be tempted to read the legacy scenarios as the spec. Out of scope (lives in `adcp-client` repo): marking the legacy `testing/scenarios/*` exports `@deprecated`, mirroring the storyboard CLI verbs, or removing the export entirely. The packaging decision is tracked in #4035; this PR ships the docs half. Co-Authored-By: Claude Opus 4.7 (1M context) --- .changeset/docs-storyboards-vs-scenarios.md | 4 ++ docs.json | 2 + docs/building/verification/conformance.mdx | 4 ++ .../verification/storyboards-vs-scenarios.mdx | 71 +++++++++++++++++++ .../verification/validate-your-agent.mdx | 4 ++ 5 files changed, 85 insertions(+) create mode 100644 .changeset/docs-storyboards-vs-scenarios.md create mode 100644 docs/building/verification/storyboards-vs-scenarios.mdx diff --git a/.changeset/docs-storyboards-vs-scenarios.md b/.changeset/docs-storyboards-vs-scenarios.md new file mode 100644 index 0000000000..506abf7a9f --- /dev/null +++ b/.changeset/docs-storyboards-vs-scenarios.md @@ -0,0 +1,4 @@ +--- +--- + +New `docs/building/verification/storyboards-vs-scenarios.mdx` disambiguating three things in AdCP that share the word "scenarios" and aren't the same: (1) **storyboards** — YAML, normative, the conformance specification; (2) **`comply_test_controller` scenarios** — protocol-level tool operations (`force_*` / `simulate_*` / `seed_*`); (3) **`@adcp/sdk/testing/scenarios/*.ts`** — legacy non-normative SDK test runners that predate `comply()`. Closes the docs half of #4035; the SDK packaging/CLI side (deprecation marker, mirrored CLI verbs, internal-only export) lives in `adcp-client`. Cross-linked from `conformance.mdx` and `validate-your-agent.mdx` callouts so readers grepping `testing/scenarios/*.ts` to learn the spec get redirected to storyboards. diff --git a/docs.json b/docs.json index 39e7628ec7..c4a26c4140 100644 --- a/docs.json +++ b/docs.json @@ -193,6 +193,7 @@ "pages": [ "docs/building/verification/conformance", "docs/building/verification/compliance-catalog", + "docs/building/verification/storyboards-vs-scenarios", "docs/building/verification/validate-your-agent", "docs/building/verification/addie-socket-mode", "docs/building/verification/grading", @@ -764,6 +765,7 @@ "pages": [ "docs/building/verification/conformance", "docs/building/verification/compliance-catalog", + "docs/building/verification/storyboards-vs-scenarios", "docs/building/verification/validate-your-agent", "docs/building/verification/grading", "docs/building/verification/get-test-ready", diff --git a/docs/building/verification/conformance.mdx b/docs/building/verification/conformance.mdx index 42e91e4e54..cf211b049c 100644 --- a/docs/building/verification/conformance.mdx +++ b/docs/building/verification/conformance.mdx @@ -38,6 +38,10 @@ Every normative rule in the suite has exactly one home: the storyboard YAML at [ This is deliberate. A separate prose spec that restates storyboard rules creates two sources of truth. Two sources of truth drift. We pick one: the suite. + +The `@adcp/sdk` package also ships TypeScript files under `testing/scenarios/` that pre-date storyboard-driven `comply()`. They are **not** the conformance spec — see [Storyboards vs. scenarios](/docs/building/verification/storyboards-vs-scenarios) for which is which. + + The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in the referenced storyboards and prose sections below are to be interpreted as described in [RFC 2119](https://www.rfc-editor.org/rfc/rfc2119). ## Conformance is layered diff --git a/docs/building/verification/storyboards-vs-scenarios.mdx b/docs/building/verification/storyboards-vs-scenarios.mdx new file mode 100644 index 0000000000..276ccf59c3 --- /dev/null +++ b/docs/building/verification/storyboards-vs-scenarios.mdx @@ -0,0 +1,71 @@ +--- +title: Storyboards vs. scenarios — which is which +sidebarTitle: Storyboards vs. Scenarios +description: "Three things in AdCP share the word 'scenarios' and they aren't the same. Here's how to tell them apart." +"og:title": "AdCP — Storyboards vs. scenarios" +--- + +If you're trying to figure out *how* to test an AdCP agent, three things in this ecosystem share overlapping vocabulary. They are not the same. Confusing them produces wrong conclusions — including the kind that get reported as protocol gaps that aren't actually gaps. + +## TL;DR + +| | What it is | Where it lives | Normative? | +|---|---|---|---| +| **Storyboards** | YAML files defining a workflow end-to-end + every wire-shape assertion. **The conformance specification.** | `/compliance/{version}/universal/*.yaml`, `protocols/*/*.yaml`, `specialisms/*/index.yaml` | ✅ Yes | +| **`comply_test_controller` scenarios** | Protocol-level tool operations a seller exposes (`force_*`, `simulate_*`, `seed_*`) so storyboards can drive deterministic state. | The `scenario` parameter of the seller's `comply_test_controller` MCP tool. | ✅ Yes | +| **`@adcp/sdk/testing/scenarios/*.ts`** | Legacy TypeScript test runners predating storyboard-driven `comply()`. | `node_modules/@adcp/sdk/dist/lib/testing/scenarios/*.js` | ❌ **No** | + +If you're reading a file at `node_modules/@adcp/sdk/dist/lib/testing/scenarios/media-buy.js` and trying to figure out what AdCP requires — **stop**. That's not the spec. Read [the storyboards](/docs/building/verification/conformance) instead. + +## Storyboards (normative) + +Storyboards are the conformance specification. Each one defines: +- A workflow (e.g. *sales-guaranteed proposal/refine/finalize lifecycle*) +- The exact tool calls a buyer agent makes +- Every wire-shape assertion the seller's responses must satisfy +- The `comply_test_controller` operations the runner uses to seed deterministic state + +[`conformance.mdx`](/docs/building/verification/conformance) calls these *"the truth"* — and means it. The AdCP Verified (Spec) badge is issued by running these YAML files through [`comply()`](/docs/building/verification/validate-your-agent) and observing every assertion pass. + +You'll see storyboards at: +- `/compliance/{version}/universal/*.yaml` — every AdCP agent must pass +- `/compliance/{version}/protocols/{protocol}/index.yaml` — anyone claiming the protocol +- `/compliance/{version}/specialisms/{id}/index.yaml` — opt-in claims + +Run them via `npx @adcp/sdk storyboard run ` or interactively through [Addie](https://agenticadvertising.org). + +## `comply_test_controller` scenarios (normative — but different) + +The word "scenario" also appears in the `comply_test_controller` MCP tool that sellers expose to support deterministic testing. Storyboards call this tool with `scenario: ` to drive seller state without waiting for real time to pass: e.g. `force_task_completion` to finish a pending async media buy, `simulate_delivery` to inject impressions, `seed_product` to install a fixture. + +These are **protocol operations**, not test runners. They're documented at [comply_test_controller](/docs/building/by-layer/L3/comply-test-controller). Sellers MUST implement them to be testable; the storyboards MUST use them to stay deterministic. + +So when you hear "the seller doesn't support that scenario," it usually means: "the seller's `comply_test_controller` doesn't expose `force_X` yet." Not "no storyboard tests X." + +## `@adcp/sdk/testing/scenarios/*.ts` (NON-normative legacy) + +The SDK ships TypeScript files under `testing/scenarios/` — `media-buy.ts`, `signals.ts`, `creative.ts`, and a dozen siblings. These predate the storyboard-driven `comply()` engine. They are: + +- **Not the conformance spec.** Reading them to learn what AdCP requires will produce wrong answers. +- **Not maintained in lockstep with storyboards.** A scenario file may hard-code parameters that the storyboard equivalent has long since corrected. *Example: `scenarios/media-buy.ts` hard-codes `buying_mode: 'brief'` at four call sites; the YAML storyboard for `sales-guaranteed` exercises `buying_mode: 'refine'` + `action: 'finalize'` via the `proposal_finalize` storyboard. Both are valid `buying_mode` values; the scenario file just doesn't cover the lifecycle.* +- **Callable, but for different purposes.** Internal smoke tests in the SDK, integration-test fixtures for downstream tooling, and some legacy paths still use these. They are not what AAO Verified (Spec) runs against. + +If you find yourself grepping `testing/scenarios/*.ts` to understand AdCP, you've taken a wrong turn. The right path: + +1. Find the storyboard that matches your declared specialism: `npx @adcp/sdk storyboard list` +2. Read the YAML: `npx @adcp/sdk storyboard show ` +3. Run it: `npx @adcp/sdk storyboard run --agent-url ` + +## Three-line decoder + +If you see… | …you're looking at | Trust as spec? +---|---|--- +A `.yaml` under `/compliance/{version}/...` | A storyboard | ✅ Yes +A seller responding to `comply_test_controller` with a `scenario` parameter | A protocol-level test-control operation | ✅ Yes (the contract is normative; the seller MAY refuse with `UNKNOWN_SCENARIO`) +A `.ts` or `.js` under `@adcp/sdk/dist/lib/testing/scenarios/` | A legacy SDK test runner | ❌ No + +## Cross-repo + +The disambiguation work in the SDK itself — marking the legacy scenarios as `@deprecated`, mirroring the storyboard CLI verbs, or removing the export entirely — lives in [`adcp-client`](https://github.com/adcontextprotocol/adcp-client). The decision on whether `testing/scenarios/*` stays public-but-deprecated or becomes internal-only is tracked in [#4035](https://github.com/adcontextprotocol/adcp/issues/4035). + +For now: if you want to test an AdCP agent, the answer is storyboards. The other two things named "scenario" have their place, but they aren't it. diff --git a/docs/building/verification/validate-your-agent.mdx b/docs/building/verification/validate-your-agent.mdx index 730935b9af..b8c7812bec 100644 --- a/docs/building/verification/validate-your-agent.mdx +++ b/docs/building/verification/validate-your-agent.mdx @@ -9,6 +9,10 @@ Once your agent is running, validate it before going live. Storyboards exercise Storyboards are available from the command line and interactively through [Addie](https://agenticadvertising.org). They are also published alongside schemas at `/compliance/{version}/` and bundled into the per-version protocol tarball at `/protocol/{version}.tgz` — see [Schemas and SDKs](/docs/building/by-layer/L0/schemas#one-shot-protocol-bundle) for how to fetch them offline. + +The `@adcp/sdk` package also exports legacy TypeScript test runners under `testing/scenarios/*` (e.g. `media-buy.ts`, `signals.ts`). These predate `comply()` and are **not** the conformance specification. If you find yourself grepping those files to learn what AdCP requires, see [Storyboards vs. scenarios](/docs/building/verification/storyboards-vs-scenarios) for which surface is normative. + + ## Storyboard taxonomy Storyboards are organized into three layers so agents declare only what they actually support: