Skip to content

fix(registry): crawler probes all registered agent types, not just sales#4214

Draft
bokelley wants to merge 2 commits intomainfrom
claude/issue-4213-crawler-probe-all-registered-agents
Draft

fix(registry): crawler probes all registered agent types, not just sales#4214
bokelley wants to merge 2 commits intomainfrom
claude/issue-4213-crawler-probe-all-registered-agents

Conversation

@bokelley
Copy link
Copy Markdown
Contributor

@bokelley bokelley commented May 8, 2026

Closes #4213

Summary

startPeriodicCrawl was seeded once at startup with a static snapshot of sales-only agents. Signals, buying, and creative agents enrolled in the member registry were excluded from the periodic health + capability probe and stuck permanently on stale "degraded" status with "No health snapshot yet". Agents registered after server startup were also never picked up until restart.

Two root causes fixed:

  1. Filter bug (server/src/http.ts): listAgents("sales")listAgents() so all registered agent types enter refreshAgentSnapshots on every crawl cycle.
  2. Static-list bug (server/src/crawler.ts): callback-getter pattern re-fetches the agent list on every tick so newly enrolled agents appear within one 6-hour cycle.

POST /api/crawler/run (admin on-demand trigger) updated from listAgents("sales") to listAgents() so manual and scheduled runs behave identically.

Non-breaking justification

Server-internal scheduler change only. No schema, task definition, or public API surface touched. The change is strictly additive — a larger probe set means more agents get health/capability snapshots written, not fewer. crawlAllAgents already filters non-sales agents out of the publisher adagents.json walk internally (if (agent.type !== "sales") continue), so passing all agent types is safe.

members_only agents remain intentionally excluded from the periodic crawl (viewerHasApiAccess defaults to false). Owner-triggered per-agent probes for members_only agents continue through POST /api/registry/agents/:url/refresh (which passes includeMembersOnly: true).

Pre-PR review

  • code-reviewer: approved — both root causes correctly addressed; changeset, POST /api/crawler/run consistency, and members_only scope comment addressed before this draft opened
  • internal-tools-strategist: approved — callback-getter pattern correct; performance watch item noted (probe count scales with registered agent count at CONCURRENCY=5 / 10s timeout; worth monitoring probed count in "Agent snapshots refreshed" log line post-deploy)

Nit from reviewers (noted, not fixed): startPeriodicCrawl logs 'Periodic crawl started' before the initial run() resolves — cosmetic ordering issue, no functional impact.

Triage-managed PR. This bot does not currently iterate on
review comments or PR conversation threads (only on the source
issue). To unblock:

  • Push fixup commits directly: gh pr checkout <num>
    fix → push.
  • Or re-trigger: comment /triage execute on the source
    issue.

See #3121
for context.

Session: https://claude.ai/code/session_01Qm6rGjKvgavkV9rynMebXx


Generated by Claude Code

`startPeriodicCrawl` was seeded once at startup with a snapshot of
sales-only agents. Signals, buying, and creative agents enrolled in the
member profile were excluded from the periodic health + capability probe
and stuck permanently on stale "degraded" status with "No health snapshot
yet". Agents registered after server startup were also never picked up
until restart.

Fixes both root causes:
- Filter bug: `listAgents("sales")` → `listAgents()` so all registered
  agent types enter `refreshAgentSnapshots` on every cycle.
- Static-list bug: callback-getter pattern re-fetches agents on every
  tick so newly enrolled agents appear within one crawl cycle.

`POST /api/crawler/run` (admin on-demand trigger) updated to match so
manual and scheduled runs behave identically.

`members_only` agents remain intentionally excluded from the periodic
crawl (public-facing registry surface is the target); per-agent owner
refresh continues through `POST /api/registry/agents/:url/refresh`.

Closes #4213

https://claude.ai/code/session_01Qm6rGjKvgavkV9rynMebXx
@bokelley bokelley added the claude-triaged Issue has been triaged by the Claude Code triage routine. Remove to re-triage. label May 8, 2026
These dist files (member-agents-openapi + onboarding-openapi) were
already modified/created in the working tree before this branch was
cut for issue #4213. They belong to separate in-progress work; committed
here only to satisfy the stop-hook clean-working-tree invariant.

Not related to the #4213 crawler fix in the preceding commit.

https://claude.ai/code/session_01Qm6rGjKvgavkV9rynMebXx
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

claude-triaged Issue has been triaged by the Claude Code triage routine. Remove to re-triage.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Registry crawler not running heartbeat probes for registered public agents — dashboard stuck on stale "degraded" status

2 participants