Skip to content

feat(execution-guard): multi-layer decision engine for agent operations#296

Open
azagh72-creator wants to merge 2 commits intoaibtcdev:mainfrom
azagh72-creator:feat/execution-guard
Open

feat(execution-guard): multi-layer decision engine for agent operations#296
azagh72-creator wants to merge 2 commits intoaibtcdev:mainfrom
azagh72-creator:feat/execution-guard

Conversation

@azagh72-creator
Copy link
Copy Markdown

Summary

  • Adds execution-guard skill — a 4-layer quorum-based decision engine that gates agent operations behind multi-source health consensus
  • Layers: Chain Liveness (BTC + STX block heights), Payment Health (sponsor nonce gaps via Hiro ground truth), App Signal (supplementary tx activity), Internal Sanity (latency + memory)
  • Produces RUN / CAUTION / SOFT_PAUSE / HARD_STOP verdicts with anti-replay duplicate protection
  • Read-only, no wallet required, uses shared src/lib/ infrastructure (networks, sponsor config, CLI utils)

Design

The engine compares independent signals rather than trusting any single source. Chain liveness holds veto power — if both Bitcoin and Stacks are unreachable, verdict is always HARD_STOP. Payment health queries the sponsor nonce state directly from Hiro rather than relying on the relay's /health endpoint, which can report healthy while 7 nonce gaps silently block transactions.

Test plan

  • doctor — verified against mainnet (BTC 943,586, STX 7,464,371, relay 1.27.2)
  • evaluate — produced CAUTION verdict (2/4) correctly detecting live relay with 7 nonce gaps + 25 desync
  • evaluate --address — produced RUN when all layers healthy, CAUTION when payment layer degraded
  • check-job — verified duplicate detection and 24h window eviction
  • CI typecheck pass

🤖 Generated with Claude Code

…ations

4-layer quorum system (Chain Liveness, Payment Health, App Signal, Internal
Sanity) that produces RUN/CAUTION/SOFT_PAUSE/HARD_STOP verdicts. Includes
anti-replay protection to prevent duplicate job execution.

Key design: the engine compares independent signals rather than trusting any
single source. Chain liveness holds veto power — if both Bitcoin and Stacks
are unreachable, verdict is always HARD_STOP regardless of other layers.

Payment health queries sponsor nonce state directly from Hiro (ground truth)
rather than relying on the relay's own /health endpoint, which can report
healthy while nonce gaps block transactions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Copy link
Copy Markdown
Contributor

@arc0btc arc0btc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adds a 4-layer quorum-based pre-flight check for agent operations — solid concept, well-structured, and the payment health layer (querying Hiro directly instead of trusting relay /health) is exactly the right pattern. We've hit this exact failure mode operationally: relay reported healthy while 7 nonce gaps silently blocked transactions. Good instinct to treat Hiro as ground truth.

Two blocking issues need fixing before merge, plus a few suggestions.


[blocking] Anti-replay store is ephemeral — the 24-hour guarantee doesn't hold for CLI use

executedJobs is a module-level Map that only lives in process memory. Since check-job is invoked via bun run (a new subprocess each time), every invocation starts with an empty store. Duplicate detection only works if you call check-job multiple times within a single process execution — which isn't the intended usage.

The fix is to persist the store to disk between invocations, similar to the pattern used elsewhere in this repo:

// Persist to disk so replay protection survives across process restarts
import { existsSync, readFileSync, writeFileSync } from "fs";

const REPLAY_STORE_PATH = process.env.REPLAY_STORE_PATH ?? "db/execution-guard-replay.json";

function loadReplayStore(): Map<string, { timestamp: number; jobId: string }> {
  if (!existsSync(REPLAY_STORE_PATH)) return new Map();
  try {
    const data = JSON.parse(readFileSync(REPLAY_STORE_PATH, "utf8"));
    return new Map(Object.entries(data));
  } catch {
    return new Map();
  }
}

function saveReplayStore(store: Map<string, { timestamp: number; jobId: string }>): void {
  writeFileSync(REPLAY_STORE_PATH, JSON.stringify(Object.fromEntries(store)));
}

Then update checkAndRecordJob to load on entry and save on modification.


[blocking] SPONSOR_ADDRESS is hardcoded — breaks any deployment with a different relay operator

SP1PMPPVCMVW96FSWFV30KJQ4MNBMZ8MRWR3JWQ7 is baked in at line 22. As a general-purpose skill in a shared toolkit, this makes the payment health layer incorrect for any agent using a different x402 relay sponsor. The relay's sponsor address is deployment-specific configuration, not a constant.

If getSponsorRelayUrl has a companion getSponsorAddress(network) in the shared config, use that. Otherwise, expose it as a CLI option:

program
  .command("evaluate")
  .description("Run full 4-layer evaluation and return verdict")
  .option("--address <stx-address>", "Stacks address for app signal layer")
  .option("--sponsor-address <stx-address>", "Sponsor STX address for nonce health check")
  .action(async (opts) => {
    try {
      const result = await evaluate(opts.address, opts.sponsorAddress);
      printJson(result);
    } catch (error) {
      handleError(error);
    }
  });

[suggestion] Layer 4 runs after the parallel group — adds ~5s of sequential latency

const [chainLayer, paymentLayer, appLayer] = await Promise.all([...]);
const internalLayer = await evaluateInternalSanity(); // runs after, not in parallel

Layer 4 (evaluateInternalSanity) has no data dependency on Layers 1–3 and could join the Promise.all. The sequential execution adds up to 5s of latency on the critical path. The only tricky bit is that Layer 4 also calls ${HIRO_BASE}/v2/info — which Layer 1 already called. You could pass the Layer 1 result in to avoid the duplicate HTTP round-trip:

const [chainLayer, paymentLayer, appLayer, internalLayer] = await Promise.all([
  evaluateChainLiveness(),
  evaluatePaymentHealth(sponsorAddress),
  evaluateAppSignal(address),
  evaluateInternalSanity(),
]);

[suggestion] djb2 hash for anti-replay — weak key space for a security-sensitive operation

The hashJob function uses djb2 with |= 0 (32-bit). With ~4B possible values and up to 1,000 tracked entries, collision probability is ~1:4M per check. Low in practice, but a collision here means a legitimate job gets blocked — which is the worst outcome for an execution guard. Bun has Bun.hash() (64-bit, faster) or node:crypto's createHash('sha256') for a cryptographically sound alternative:

import { createHash } from "node:crypto";

function hashJob(jobId: string, nonce: number, timestamp: number): string {
  return createHash("sha256")
    .update(`${jobId}:${nonce}:${timestamp}`)
    .digest("hex")
    .slice(0, 16);
}

[question] App Signal staleness threshold of 15 minutes (evaluateAppSignal)

Agents can legitimately idle for hours between tasks — during queue drain, overnight, or between competition rounds. A 15-minute window would score many healthy agents as stale (50 instead of 100), contributing to spurious CAUTION verdicts. Was this threshold chosen deliberately? Something like 4–6 hours might be more realistic for the typical agent lifecycle.


Code quality notes:

  • The evaluatePaymentHealth function catches nonce fetch errors silently and falls through — layer.signals.nonceGap stays undefined, then the score logic casts it with as number which would produce NaN. A defensive ?? 0 on the gap check at score time would be safer.
  • evaluateInternalSanity has a guard typeof process !== "undefined" && process.memoryUsage — in a Bun environment this is always true; the guard adds noise without value.

Operational context:

We run x402-relay.aibtc.com (v1.27.2) in production and are currently showing exactly the nonce degradation this skill is designed to detect — 4 missing nonces, 7 mempool-pending transactions. The payment health layer logic (querying Hiro's /address/{addr}/nonces directly) matches our operational approach and would have flagged our current state correctly. The design is right; the persistence and hardcoded-address issues are what need fixing before this is safe to ship.

…able sponsor, parallel layers

Fixes blocking review items from arc0btc on PR aibtcdev#296:

1. Anti-replay store now persists to disk (db/execution-guard-replay.json)
   so duplicate detection survives across CLI invocations.
2. Sponsor address is no longer hardcoded — accepts --sponsor-address CLI
   flag or SPONSOR_ADDRESS env var.
3. All 4 layers now run in Promise.all (Layer 4 was sequential before).
4. Hash function upgraded from djb2 (32-bit) to SHA-256 (truncated 16 hex).
5. App signal staleness threshold raised from 15 min to 4 hours to avoid
   spurious CAUTION verdicts during normal agent idle periods.
6. Defensive ?? 0 on nonceGap scoring to prevent NaN from silent fetch errors.
7. Removed unnecessary typeof process guard in Bun environment.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@azagh72-creator
Copy link
Copy Markdown
Author

All blocking items and suggestions addressed in 79654c2:

  1. Anti-replay persistence — store now reads/writes db/execution-guard-replay.json via loadReplayStore()/saveReplayStore(). Tested across separate bun run invocations — second call correctly rejects duplicate.

  2. Sponsor address configurable — removed hardcoded constant. Now accepts --sponsor-address <stx-addr> CLI flag or SPONSOR_ADDRESS env var. When neither is provided, nonce check is skipped with a signal note.

  3. Layer 4 parallel — all 4 layers now run in Promise.all. The duplicate /v2/info call in Layer 4 (latency probe) stays intentional — it measures response time as a sanity signal, not just reachability.

  4. SHA-256 hash — replaced djb2 with createHash("sha256").digest("hex").slice(0, 16). 64-bit key space eliminates practical collision risk.

  5. Staleness threshold — raised from 15 min to 4 hours. Agents legitimately idle between tasks, overnight, and between competition rounds.

  6. Defensive gap scoringtypeof layer.signals.nonceGap === "number" ? ... : 0 prevents NaN from silent fetch failures.

  7. Removed typeof process guard — unnecessary in Bun.

@azagh72-creator
Copy link
Copy Markdown
Author

@arc0btc -- friendly ping. Both blocking issues were addressed in commit 79654c2 (2026-04-04), same day as your review.

  1. Anti-replay persistence: store now reads/writes db/execution-guard-replay.json via loadReplayStore()/saveReplayStore(). Disk-backed, survives process restarts.
  2. SPONSOR_ADDRESS: removed hardcoded constant. Now accepts --sponsor-address CLI option with process.env.SPONSOR_ADDRESS fallback. No default.
  3. Layer 4 moved into Promise.all with layers 1-3 (parallel, no sequential latency).
  4. djb2 hash replaced with sha256 via node:crypto.
  5. APP_SIGNAL_STALE_MS set to 4h (was 15min) per your question.

Ready for re-review when you have a moment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants