Skip to content

fix(ci): bump AVM check-circuit per-tx timeout from 30s to 90s#23330

Draft
AztecBot wants to merge 1 commit into
nextfrom
claudebox/avm-check-circuit-fix
Draft

fix(ci): bump AVM check-circuit per-tx timeout from 30s to 90s#23330
AztecBot wants to merge 1 commit into
nextfrom
claudebox/avm-check-circuit-fix

Conversation

@AztecBot
Copy link
Copy Markdown
Collaborator

Summary

The nightly AVM Circuit Inputs Collection and Check workflow run 25952408817 failed in the avm-check-circuit job with Process completed with exit code 124 — the standard timeout(1) SIGTERM exit. No constraint failures; one of the bb-avm avm_check_circuit invocations just took longer than the 30s budget.

Root cause

avm_check_circuit_cmds in yarn-project/end-to-end/bootstrap.sh emits one parallelize command per dumped tx with TIMEOUT=30s. The avm_cc_* tests have no owners entry in .test_patterns.yml, so a single timeout hard-fails the job — no retry.

Reproduced locally against the same commit (downloaded the public S3 artifact tarball, built bb-avm, ran every input):

  • 615/617 inputs finish in ≤ 2s.
  • e2e_storage_proof heaviest tx: 10s.
  • e2e_multiple_blobs/...0x0b6dc9867b...bin (the emit_full_size_public_log batch tx): 23s wall on a 186-core c7a, 21.5s wall (43.6s CPU) with taskset -c 0-3 to simulate the 4-vCPU ubuntu-latest runner.

That single tx was always sitting right against the 30s line. It finally tipped over on a slow runner.

Fix

Raise TIMEOUT to 90s, ~4× the current slow tx's wall time on a 4-vCPU runner. That absorbs runner variance and modest future AVM trace growth while still killing a genuinely hung run quickly.

Full analysis (numbers, per-stage tracegen breakdown, repro steps): https://gist.github.com/AztecBot/a47798ee8721db19667e9b8844f86b3a

ClaudeBox log: https://claudebox.work/s/27ac153ca9b8a375?run=1

@AztecBot AztecBot added ci-draft Run CI on draft PRs. claudebox Owned by claudebox. it can push to this PR. labels May 16, 2026
@AztecBot
Copy link
Copy Markdown
Collaborator Author

Flakey Tests

🤖 says: This CI run detected 1 tests that failed, but were tolerated due to a .test_patterns.yml entry.

\033FLAKED\033 (8;;http://ci.aztec-labs.com/9df0e24d9b6318d2�9df0e24d9b6318d28;;�):  yarn-project/end-to-end/scripts/run_test.sh simple src/e2e_p2p/duplicate_proposal_slash.test.ts (263s) (code: 0) group:e2e-p2p-epoch-flakes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-draft Run CI on draft PRs. claudebox Owned by claudebox. it can push to this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant