Registry: remove `discovered` agents entirely — registered-only public surface

[EmmaLouise2018]

Registry: remove discovered agents entirely — registered-only public surface

Refs #3538 (Problem 3, Problem 6), #3547, #3573, #3495.

Proposal

Stop surfacing source: 'discovered' agents in the AAO public registry. The registry should reflect attested, opt-in agents only. Crawled-from-adagents.json agents that have not enrolled are not attested by their owner and should not be displayed, filtered on, or returned by the public API.

Why now

We have spent the last month patching the seams of registered-vs-discovered conflation:

• Registry: follow-ups from #3495 — stale types, registered/discovered conflation, docs IA gaps, non-member path #3538 Problem 3 — registered + discovered are merged in /api/registry/agents. Public UI does not segment them. Shipped: ?source= query param, schema annotations, doc copy.
• decision(registry): how should discovered agents surface member-publisher endorsement (#3538 Problem 6) #3547 / Problem 6 — discovered agents always rendered member: null even when the publisher domain is owned by an AAO member. Solution shipped: a new endorsed_by_publisher_member field. Net effect: more schema, more docs, more UI, to communicate a partial trust signal.
• fix(registry): canonicalize agent_url before registered/discovered collapse (trailing slash + scheme) #3573 — registered/discovered collapse breaks on trailing-slash + scheme variance.
• Sales agents discovered via adagents.json registered as type:'buying' instead of 'sales' #3495 — sales agents discovered via adagents.json were registered as type: 'buying'. The discovered path is the original source of the type-inference mess that fix(registry): flip type==='buying' filter inversions to 'sales' + add sales to by_type tally #3540/ops(registry): backfill stale member_profiles.agents types + crawler reclassify-on-disagreement #3541 just finished cleaning up.

Every one of these is a tax on a feature that no member asked for. Discovered agents are a crawler artifact, not a product. The registry's contract with members is "you enrolled, you are listed." A crawler-only entry breaks that contract from both sides — the listed agent never opted in, and the registry consumer can't tell what the entry means without reading three doc pages.

Scope of removal

Public API

• /api/registry/agents — return only registered agents. Drop source, discovered_from, endorsed_by_publisher_member from FederatedAgentWithDetails.
• /api/registry/agents/:url — 404 for URLs that only exist via discovery.
• ?source= query param — remove (introduced for Problem 3 step 2).
• Same treatment for FederatedPublisher — registered publishers only.

Schema / types

• server/src/schemas/registry.ts:432-484 — drop the ["registered", "discovered"] enum, DiscoveredFromSchema, EndorsedByPublisherMemberSchema.
• server/src/types.ts:1061-1108 — FederatedAgent.source, discovered_from, endorsed_by_publisher_member; FederatedPublisher.source, discovered_from.

Service layer

• server/src/federated-index.ts:30-140 — listAllAgents() no longer merges; returns registered only. Delete the dedup-by-canonical-URL logic that fix(registry): canonicalize agent_url before registered/discovered collapse (trailing slash + scheme) #3573 was about to land.
• server/src/db/federated-index-db.ts:378-505 — drop upsertDiscoveredAgent, getDiscoveredAgents, related update/expire helpers.

Crawler

• server/src/crawler.ts — keep adagents.json parsing for the publisher-authorization graph (we still need to know who endorses whom for list_authorized_properties validation), but stop registering unknown agents from those listings. The crawler becomes a verification tool for known agents, not a discovery tool for new ones.
• stats.discovered_agents (line 472) — remove.

Database

• discovered_agents table — drop.
• discovered_publishers table — drop, or preserve as crawled_publishers if any publisher-graph code still needs the row (decide during impl).
• agent_publisher_authorizations — keep the table (authorizations still matter), but the source: 'agent_claim' | 'adagents_json' enum stays meaningful only as an audit field, not as a registry-source signal.
• New migration to drop tables + clean up indices + remove cleanup/expiry jobs from migrations 331, 349, 432, 455, 461.

UI

• server/public/agents.html — remove any source badge/filter/segment.
• /registry page — single list, no tabs.

Docs

• docs/registry/registering-an-agent.mdx — collapse the "four crawl paths" framing to "one path: enroll on your member profile."
• docs/registry/index.mdx overview — drop the registered-vs-discovered explainer.
• Remove endorsed_by_publisher_member and discovered_from references throughout.
• Mintlify example responses — simpler, no per-source overrides.

Tests

• Delete: server/tests/unit/registry-discovered-endorsement.test.ts, the discovered branches of registry-sync tests.
• Update: any test asserting on source or merged-list behavior.

What we lose, and why it's fine

1. Visibility into "agents that exist in the wild but haven't joined." Real, but the registry was never the right surface for this. If we want a market-coverage view, build it as an internal ops tool, not a public registry tab.
2. Publisher endorsement signal (endorsed_by_publisher_member). Shipped two days ago in decision(registry): how should discovered agents surface member-publisher endorsement (#3538 Problem 6) #3547, deleted here. Net: the right move. The signal was a workaround for a category we are now removing.
3. Crawler-driven onboarding funnel. A discovered listing was an implicit nudge to the agent owner to enroll. We can replace this with an outbound email when the crawler sees a new adagents.json entry pointing at an unregistered URL — a stronger nudge than "you're already in the registry, ish."
4. Backwards compatibility for any external consumer that filtered on ?source=discovered. The ?source= param is days old; risk is near zero.

Migration plan

1. PR 1 — feature flag the public surface. Add REGISTRY_HIDE_DISCOVERED=true (default true in prod) that filters discovered out of /api/registry/agents + UI. No schema or DB changes. Ship behind flag, validate.
2. PR 2 — remove the type fields. Drop source / discovered_from / endorsed_by_publisher_member from response schemas and TS types. Update OpenAPI / docs.
3. PR 3 — service + DB cleanup. Strip federated-index.ts merge logic. Drop discovered_agents / discovered_publishers tables in a migration. Repoint crawler to verification-only mode.
4. PR 4 — docs sweep. Collapse registering-an-agent docs, remove explainers, fix examples.

Open questions

• Do we keep agent_publisher_authorizations populated from crawl? Probably yes — list_authorized_properties validation still depends on it. The change is "we record the relationship; we do not turn the relationship into a registry entry."
• Outbound nudge? Worth it as a follow-up issue, not a blocker on this one.
• Compliance suite (AAO discovery crawler has no response-body byte cap (DoS via hostile vendor) #3731-adjacent). Crawler still needs the byte-cap fix from AAO discovery crawler has no response-body byte cap (DoS via hostile vendor) #3731 regardless of this issue; mention as adjacent, not blocking.

Acceptance

• count(*) on /api/registry/agents matches count(*) on member_profiles.agents after dedup.
• Zero references to discovered, discovered_from, endorsed_by_publisher_member outside of migration history.
• discovered_agents and discovered_publishers tables are dropped (or renamed to reflect their crawler-graph-only role).
• Docs no longer mention registered-vs-discovered.

[bokelley]

Triage

Classification: Feature request (architectural removal)
Bucket(s): registry / discovery, admin / ops tools
Status: ready-for-human

What the experts said:

• ad-tech-protocol-expert: Industry precedent (sellers.json, OAuth, IANA) supports registered-only surfaces; removing source/discovered_from/endorsed_by_publisher_member is a MAJOR semver bump per repo policy, and dropping discovered_agents mid-cutover while Unify property registry: retire discovered_properties, single source of truth in catalog #3177 PR 5 (UNION cutover) is still open would break the publisher-lookup paths today.
• adtech-product-expert: The "no member asked for it" claim holds for buy-side; the real risk is the onboarding funnel — removing the passive discovered entry with no outbound nudge in place creates a window where legitimate sell-side agents simply vanish from the registry.
• internal-tools-strategist: Two runtime-crash blockers the migration plan doesn't address: (1) migration 349 puts agent_inventory_profiles.agent_url → discovered_agents(agent_url) ON DELETE CASCADE — dropping the table silently deletes all inventory profile rows; (2) the deleteExpired cleanup job queries discovered_agents at runtime and will crash after the drop unless patched in PR 2, not PR 3. Also: recordAgentFromAdagentsJson couples the discovery write and authorization write — keeping one requires splitting the method before PR 2 changes the types.

My take: The architectural direction is correct — a public protocol registry should surface only attested, opt-in agents, matching sellers.json/OAuth precedent. But this is a breaking API change (fields removed, 404s introduced, ?source= removed) with two runtime-crash blockers and an active PR conflict to resolve first. @bokelley, your call: Option A (merge #3773 + #3539, accept the seam-patching model) or Option B (close #3773 + #3539, adopt the removal plan with the pre-requisite fixes below)?

---

Option A — Fix the seams; merge #3773 and keep discovered:

 // GET /api/registry/agents — stays mixed
   { "agent_url": "https://acme-ads.example.com/",
     "source": "registered",
     "member": { "name": "Acme Corp" } }
+  { "agent_url": "https://crawled.example.com/",
+    "source": "discovered",
+    "discovered_from": "https://…/.well-known/adagents.json",
+    "endorsed_by_publisher_member": true,
+    "member": null }

PR #3773 merges (canonicalization fix ships). PR #3539 merges (docs/annotations ship). This issue closes as won't-fix.

Option B — Remove discovered; close #3773 + #3539, but fix blockers first:

 // GET /api/registry/agents — registered-only
   { "agent_url": "https://acme-ads.example.com/",
     "member": { "name": "Acme Corp" } }
-  // crawled-only agents → 404 on lookup, absent from list
-  // ?source= param → removed
-  // source / discovered_from / endorsed_by_publisher_member → dropped

Pre-requisites before PR 3 (DB drop):

1. Re-key or drop agent_inventory_profiles FK before discovered_agents is dropped (migration 349 cascade)
2. Patch deleteExpired runtime query in PR 2, not PR 3
3. Sequence after Unify property registry: retire discovered_properties, single source of truth in catalog #3177 PR 5 (UNION cutover) — removing mid-cutover breaks hasValidAdagents publisher lookups
4. Confirm adagents.json schema makes no normative "will appear in registry" claim (expert flag — could widen the breaking-change scope)
5. Ship outbound nudge campaign alongside or before removal to avoid sell-side coverage gap

Note: agent_publisher_authorizations correctly stays and the crawler becomes verification-only — that part of the design is sound.

---

Triaged by Claude Code. Session: https://claude.ai/code/session_017difC755JzDi9io2txnM7D

---

Generated by Claude Code

[EmmaLouise2018]

Status — partial ship in #3780

#3780 ships the schema/type/service-layer half of this issue. Not closing yet because two follow-ups remain.

Shipped in #3780

• source / discovered_from / endorsed_by_publisher_member / discovered_at removed from response schemas + TS types
• ?source= query param removed
• sources: { registered, discovered } count blocks removed
• listAllAgents / listAllPublishers return registered-only (merge logic stripped)
• Per-agent source label removed from lookupDomain / DomainLookupResult
• 2 obsolete unit tests deleted, 2 baseline integration tests updated to pin the new contract

Remaining work — tracked here, NOT split into separate issues

Follow-up A — drop discovered_agents / discovered_publishers tables + finish crawler repoint. Two pre-requisites flagged by triage that this PR does not address:

1. Migration 349 has agent_inventory_profiles.agent_url → discovered_agents(agent_url) ON DELETE CASCADE — dropping the table will silently delete every inventory profile row. Re-key or drop the FK before the table-drop migration.
2. deleteExpired (and any other runtime query against discovered_agents / discovered_publishers) crashes after the table-drop unless patched in the same PR.

Sequencing note: do not land before #3177 PR 5 (UNION cutover) — hasValidAdagents reads from discovered_publishers during the dual-read window.

Follow-up B — docs sweep. docs/registry/registering-an-agent.mdx and adjacent MDX still describe the registered-vs-discovered distinction. Mintlify auto-populated examples need per-source overrides removed. The "four crawl paths" framing in the registering-an-agent page collapses to "one path: enroll on your member profile."

Acceptance — unchanged from original

• count(*) on /api/registry/agents matches count(*) on member_profiles.agents after dedup ✅ (feat(registry): remove discovered agents from public surface #3780)
• Zero references to discovered, discovered_from, endorsed_by_publisher_member outside migration history — partially done; remaining hits are in docs/, the DB layer's internal types, and the crawler write path (intentional, until follow-up A)
• discovered_agents / discovered_publishers tables dropped — pending follow-up A
• Docs no longer mention registered-vs-discovered — pending follow-up B