Skip to content

[DO-NOT-MERGE] execution: hive eest ci to include all devnet shards#21404

Open
taratorio wants to merge 7 commits into
mainfrom
worktree-hive-eest-full-devnet-shards
Open

[DO-NOT-MERGE] execution: hive eest ci to include all devnet shards#21404
taratorio wants to merge 7 commits into
mainfrom
worktree-hive-eest-full-devnet-shards

Conversation

@taratorio
Copy link
Copy Markdown
Member

Why

The previous glamsterdam-devnet shard filtered --sim.limit by the specific BAL-era EIP numbers (7708|7778|7843|7928|7954|7976|7981|8024|8037). That kept the shard fast, but it skipped every test in the eest_devnet fixtures that wasn't on that EIP list — including ~19k regenerated previous-fork tests (Paris/Shanghai/Cancun/Prague/Osaka) that exist precisely to verify those forks still pass under Amsterdam-era state changes. We were not exercising any of them in CI.

This PR closes that gap: full per-fork coverage of eest_devnet, every shard under the 30-min CI budget.

What

Replace the single EIP-filtered glamsterdam-devnet shard with 7 devnet shards mirroring the eest_stable per-fork split, plus two splits forced by what we found while bringing it up:

Shard sim-limit Tests Failed Duration
paris-devnet .*/.*fork_Paris 2,744 0 11m 35s
shanghai-devnet .*/.*fork_Shanghai 3,001 0 7m 52s
cancun-devnet .*/.*fork_Cancun 12,362 0 17m 59s
prague-devnet .*/.*fork_Prague 16,040 0 24m 03s
osaka-devnet .*/.*fork_Osaka 16,750 0 26m 06s
amsterdam-a-l-devnet .*tests/[a-l].*fork_Amsterdam 11,096 0* 23m 31s
amsterdam-m-z-devnet .*tests/[m-z].*fork_Amsterdam ~10,291 est. 0 ~25 min (est.)
total 72,284 1 (flake) ~2h 17m (cumulative)

Measured locally against bal@v7.2.0 / devnets/bal/7, shards run sequentially. In CI they run in parallel, so wall time is bounded by the longest shard (~26 min).

* amsterdam-a-l-devnet was verified separately with 0 failures, but the EIP-7928 test_bal_invalid_extraneous_entries flake (#21364) fired once during the combined fork_Amsterdam run that produced the totals row. The flake lives in tests/amsterdam/... so it falls in the [a-l] half — hence the max-failures: 1 budget there.

Two splits worth calling out

  1. paris-devnet and shanghai-devnet are separate shards (instead of one paris+shanghai-devnet mirroring stable). The consume-engine plugin on devnets/bal/7 does not honour regex alternation in --sim.limit: empirically, ".*/.*fork_(Paris|Shanghai)" leaks ~5.8k fork_Cancun tests into the shard (verified by inspecting the testCases JSON). Single-fork patterns filter correctly. Worth chasing upstream, but splitting unblocks us today.

  2. fork_Amsterdam is split into two shards by first-directory letter under tests/ ([a-l] vs [m-z]). The combined shard ran ~21k tests in ~52 min, over the 30-min target. The letter split gives a ~52/48 partition; both halves measured under 26 min locally. Character-class regex was verified locally to filter exactly (no leakage between halves).

Failure budget

amsterdam-a-l-devnet carries max-failures: 1 for the EIP-7928 test_bal_invalid_extraneous_entries flake (#21364) — the test lives in tests/amsterdam/... so it falls in the [a-l] half. Every other devnet shard is max-failures: 0.

Docs

.claude/skills/hive-test/SKILL.md updated to match the new shard list, max-failures table, and the alternation/letter-split rationale.

@taratorio
Copy link
Copy Markdown
Member Author

let's wait for 1 Hive EEST job run before merging: https://github.com/erigontech/erigon/actions/runs/26396228185

@taratorio taratorio changed the title execution: hive eest ci to include all devnet shards [DO-NOT-MERGE] execution: hive eest ci to include all devnet shards May 25, 2026
@taratorio taratorio changed the title [DO-NOT-MERGE] execution: hive eest ci to include all devnet shards execution: hive eest ci to include all devnet shards May 25, 2026
@taratorio
Copy link
Copy Markdown
Member Author

let's wait for 1 Hive EEST job run before merging: https://github.com/erigontech/erigon/actions/runs/26396228185

all green in 31mins, should be good to go

@taratorio taratorio changed the title execution: hive eest ci to include all devnet shards [DO-NOT-MERGE] execution: hive eest ci to include all devnet shards May 25, 2026
@taratorio
Copy link
Copy Markdown
Member Author

let's wait for 1 Hive EEST job run before merging: https://github.com/erigontech/erigon/actions/runs/26396228185

all green in 31mins, should be good to go

2nd run had a flake, will investigate first before merging, let's hold off

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants