Skip to content

Commit 5e3faa4

Browse files
bloveclaudegithub-actions[bot]
authored
ci: shard examples/chat e2e into 4 parallel matrix jobs (#521)
* docs(spec): examples/chat e2e matrix sharding design Goal: cut wall-time of `examples/chat — e2e` by running it across a 4-way matrix using Playwright's built-in --shard=N/M flag. Mirrors the cockpit fleet's matrix philosophy without restructuring the underlying e2e harness. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(spec): refine e2e shard design — numeric matrix, artifact naming Self-review fixes: - Matrix uses numeric `shard: [1, 2, 3, 4]` rather than `'1/4'` strings, because GitHub Actions artifact names cannot contain `/`. - Explicit `--skip-nx-cache` preservation + `--` separator placement. - Trace artifact name templated to avoid same-name collision across matrix jobs (actions/upload-artifact@v4 errors on identical names). - Required-checks list spelled out by full check name. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(plan): examples/chat e2e matrix sharding implementation plan Two-task plan: (1) edit ci.yml's examples-chat-e2e job to use a 4-way shard matrix + Playwright --shard flag + templated artifact name; (2) push, open PR, verify per-shard wall-time on first CI run. Spec: docs/superpowers/specs/2026-05-21-examples-chat-e2e-shard-design.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ci: shard examples/chat e2e into 4 parallel matrix jobs Playwright's --shard=N/M flag auto-distributes the 18 spec files across 4 GitHub Actions matrix entries. Mirrors the cockpit fleet's matrix philosophy. Expected wall-time: ~6 min → ~2-3 min. - strategy.matrix.shard: [1, 2, 3, 4], fail-fast: false - name templated: examples/chat — e2e (1/4), (2/4), ... - --shard=N/4 passed through Nx executor's -- separator - Trace artifact templated examples-chat-e2e-trace-shard-N to avoid same-name collision in actions/upload-artifact@v4 Downstream needs: references (release aggregator, demo-deploy gate) unchanged — GitHub aggregates matrix results automatically. POST-MERGE ACTION ITEM: update GitHub branch protection required-checks list from 'examples/chat — e2e' to the 4 templated names. Spec: docs/superpowers/specs/2026-05-21-examples-chat-e2e-shard-design.md * chore(docs): regenerate api docs --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
1 parent caa995b commit 5e3faa4

3 files changed

Lines changed: 381 additions & 3 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -240,11 +240,15 @@ jobs:
240240
- run: npx nx run examples-chat-python:smoke --skip-nx-cache
241241

242242
examples-chat-e2e:
243-
name: examples/chat — e2e
243+
name: "examples/chat — e2e (${{ matrix.shard }}/4)"
244244
needs: ci-scope
245245
if: github.event_name == 'push' || needs.ci-scope.outputs.examples_chat == 'true'
246246
runs-on: ubuntu-latest
247247
timeout-minutes: 35
248+
strategy:
249+
fail-fast: false
250+
matrix:
251+
shard: [1, 2, 3, 4]
248252
steps:
249253
- uses: actions/checkout@v6.0.2
250254
- uses: actions/setup-node@v6.3.0
@@ -259,12 +263,12 @@ jobs:
259263
- working-directory: examples/chat/python
260264
run: uv sync
261265
- run: npx playwright install --with-deps chromium
262-
- run: npx nx e2e examples-chat-angular --skip-nx-cache
266+
- run: npx nx e2e examples-chat-angular --skip-nx-cache -- --shard=${{ matrix.shard }}/4
263267
- name: Upload Playwright trace on failure
264268
if: failure()
265269
uses: actions/upload-artifact@v4
266270
with:
267-
name: examples-chat-e2e-trace
271+
name: examples-chat-e2e-trace-shard-${{ matrix.shard }}
268272
path: |
269273
test-results/
270274
examples/chat/angular/e2e/test-results/

‎docs/superpowers/plans/2026-05-21-examples-chat-e2e-shard.md‎

Lines changed: 267 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,267 @@
1+
# examples/chat e2e — matrix sharding Implementation Plan
2+
3+
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
4+
5+
**Goal:** Cut wall-time of the `examples/chat — e2e` CI job by running it across a 4-way matrix using Playwright's built-in `--shard=N/M` flag.
6+
7+
**Architecture:** A single change site (`.github/workflows/ci.yml`): convert the `examples-chat-e2e` job from a single runner to a `matrix.shard: [1, 2, 3, 4]` job that forwards `--shard=N/4` to Playwright. Each shard runs ~8 of the 31 tests across the 19 spec files. Per-shard trace artifact is renamed to include the shard index to avoid name collisions in `actions/upload-artifact@v4`. Downstream jobs that `need:` this job (`release-please` aggregator at line 374, `demo-deploy` at line 544) continue to work as-is because GitHub aggregates matrix job results: `needs.examples-chat-e2e.result == 'success'` only if all 4 shards succeed.
8+
9+
**Tech Stack:** GitHub Actions, Playwright (`@playwright/test`), Nx (`@nx/playwright:playwright` executor).
10+
11+
**Out of scope** (per spec): harness consolidation, retry tuning, `fullyParallel: true` within a shard, timeout retuning, branch-protection rule updates (done by repo admin post-merge).
12+
13+
---
14+
15+
### Task 1: Convert `examples-chat-e2e` to a 4-way shard matrix
16+
17+
**Files:**
18+
- Modify: `.github/workflows/ci.yml:242-271`
19+
20+
- [ ] **Step 1: Read the current job text to confirm starting state**
21+
22+
Run: `sed -n '242,271p' .github/workflows/ci.yml`
23+
24+
Expected output begins with ` examples-chat-e2e:` and ends with the artifact `retention-days: 7` line. The current run command is:
25+
26+
```
27+
- run: npx nx e2e examples-chat-angular --skip-nx-cache
28+
```
29+
30+
The current artifact name is `examples-chat-e2e-trace`.
31+
32+
If the output differs, stop and reconcile against the spec at `docs/superpowers/specs/2026-05-21-examples-chat-e2e-shard-design.md` before editing.
33+
34+
- [ ] **Step 2: Apply the matrix + shard-flag + templated name + unique artifact**
35+
36+
Replace the block at `.github/workflows/ci.yml:242-271` with:
37+
38+
```yaml
39+
examples-chat-e2e:
40+
name: "examples/chat — e2e (${{ matrix.shard }}/4)"
41+
needs: ci-scope
42+
if: github.event_name == 'push' || needs.ci-scope.outputs.examples_chat == 'true'
43+
runs-on: ubuntu-latest
44+
timeout-minutes: 35
45+
strategy:
46+
fail-fast: false
47+
matrix:
48+
shard: [1, 2, 3, 4]
49+
steps:
50+
- uses: actions/checkout@v6.0.2
51+
- uses: actions/setup-node@v6.3.0
52+
with:
53+
node-version: 22
54+
cache: npm
55+
- name: Install uv
56+
uses: astral-sh/setup-uv@v8.0.0
57+
with:
58+
python-version: '3.12'
59+
- run: npm ci
60+
- working-directory: examples/chat/python
61+
run: uv sync
62+
- run: npx playwright install --with-deps chromium
63+
- run: npx nx e2e examples-chat-angular --skip-nx-cache -- --shard=${{ matrix.shard }}/4
64+
- name: Upload Playwright trace on failure
65+
if: failure()
66+
uses: actions/upload-artifact@v4
67+
with:
68+
name: examples-chat-e2e-trace-shard-${{ matrix.shard }}
69+
path: |
70+
test-results/
71+
examples/chat/angular/e2e/test-results/
72+
retention-days: 7
73+
```
74+
75+
Three substantive changes vs. the prior block:
76+
1. `name:` is quoted and templated with `${{ matrix.shard }}/4` so each matrix entry surfaces a distinct check name like `examples/chat — e2e (1/4)`.
77+
2. New `strategy:` block above `steps:` with `fail-fast: false` and `matrix: { shard: [1, 2, 3, 4] }`.
78+
3. The `npx nx e2e` line forwards `--shard=N/4` to Playwright after the Nx executor's `--` separator. `--skip-nx-cache` is preserved.
79+
4. Artifact `name:` includes `-shard-${{ matrix.shard }}` — required because `actions/upload-artifact@v4` errors on duplicate artifact names across matrix entries.
80+
81+
- [ ] **Step 3: Run a YAML syntax check**
82+
83+
Run: `python3 -c "import yaml; yaml.safe_load(open('.github/workflows/ci.yml'))"`
84+
85+
Expected: no output, exit code 0.
86+
87+
If a `ScannerError` or `ParserError` fires, the most likely cause is indentation drift on the `strategy:` block. Compare against the cockpit-e2e job at `.github/workflows/ci.yml:273` for a reference matrix-strategy shape that already parses.
88+
89+
- [ ] **Step 4: Confirm the downstream `needs:` references are still valid**
90+
91+
Run: `grep -n 'examples-chat-e2e' .github/workflows/ci.yml`
92+
93+
Expected: 4 matches —
94+
- Line ~242: the job definition (` examples-chat-e2e:`)
95+
- Line ~374: inside the aggregator job's `needs: [...]` list
96+
- Line ~544: inside `demo-deploy`'s `needs: [examples-chat-smoke, examples-chat-e2e]`
97+
- Line ~554: inside `demo-deploy`'s gate `if [ "${{ needs.examples-chat-e2e.result }}" != "success" ]`
98+
99+
No changes to those four sites: GitHub aggregates matrix job results — `needs.examples-chat-e2e.result` is `success` iff every shard succeeds, `failure` if any shard fails.
100+
101+
If the line numbers have drifted (e.g. another edit landed first), update this checklist's expectations but do NOT modify those lines.
102+
103+
- [ ] **Step 5: Local dry-run — verify shard distribution**
104+
105+
A local Playwright `--list-files` confirms that the 4 shards collectively cover the 19 spec files with no overlap.
106+
107+
Run (from repo root, after `npm ci` if not already done):
108+
109+
```bash
110+
for n in 1 2 3 4; do
111+
echo "=== shard $n/4 ==="
112+
npx playwright test --config=examples/chat/angular/e2e/playwright.config.ts \
113+
--shard=$n/4 --list 2>/dev/null \
114+
| grep -E '\.spec\.ts' | sed 's/^ *//' | sort -u
115+
done
116+
```
117+
118+
Expected:
119+
- Each shard prints a non-empty list of spec files.
120+
- Union across all 4 shards has 19 unique paths (matches `ls examples/chat/angular/e2e/*.spec.ts | grep -v aimock-runner | wc -l`).
121+
- Intersection between any two shards is empty.
122+
123+
Quick union sanity check in one line:
124+
125+
```bash
126+
for n in 1 2 3 4; do npx playwright test --config=examples/chat/angular/e2e/playwright.config.ts --shard=$n/4 --list 2>/dev/null; done \
127+
| grep -oE 'examples/chat/angular/e2e/[a-z-]+\.spec\.ts' | sort -u | wc -l
128+
```
129+
130+
Expected output: `19` (the count of `.spec.ts` files under that directory excluding `aimock-runner.spec.ts`, which is excluded by `testIgnore` in playwright.config.ts).
131+
132+
If the count differs from 19, do NOT proceed — investigate. The most likely cause is that a new spec was added that Playwright doesn't include in any shard (unlikely with `--shard` default behavior, but possible if `testMatch` excludes it).
133+
134+
- [ ] **Step 6: Commit**
135+
136+
```bash
137+
git add .github/workflows/ci.yml
138+
git commit -m "ci: shard examples/chat e2e into 4 parallel matrix jobs
139+
140+
Playwright's --shard=N/M flag auto-distributes the 19 spec files
141+
across 4 GitHub Actions matrix entries. Mirrors the cockpit fleet's
142+
matrix philosophy. Expected wall-time: ~6 min → ~2-3 min.
143+
144+
- strategy.matrix.shard: [1, 2, 3, 4], fail-fast: false
145+
- name templated: examples/chat — e2e (1/4), (2/4), ...
146+
- --shard=N/4 passed through Nx executor's -- separator
147+
- Trace artifact templated examples-chat-e2e-trace-shard-N to avoid
148+
same-name collision in actions/upload-artifact@v4
149+
150+
Downstream needs: references (release aggregator, demo-deploy gate)
151+
unchanged — GitHub aggregates matrix results automatically.
152+
153+
POST-MERGE ACTION ITEM: update GitHub branch protection required-checks
154+
list from 'examples/chat — e2e' to the 4 templated names.
155+
156+
Spec: docs/superpowers/specs/2026-05-21-examples-chat-e2e-shard-design.md
157+
"
158+
```
159+
160+
---
161+
162+
### Task 2: Push, open PR, verify first CI run
163+
164+
**Files:**
165+
- None modified (CI observation + PR metadata only)
166+
167+
- [ ] **Step 1: Push the branch**
168+
169+
Run: `git push -u origin claude/chat-e2e-shard`
170+
171+
Expected: push succeeds, branch tracked.
172+
173+
- [ ] **Step 2: Open the PR**
174+
175+
Run:
176+
177+
```bash
178+
gh pr create --title "ci: shard examples/chat e2e into 4 parallel matrix jobs" --body "$(cat <<'EOF'
179+
## Summary
180+
181+
- Convert the singular \`examples/chat — e2e\` CI job into a 4-way matrix using Playwright's built-in \`--shard=N/M\` flag.
182+
- Mirrors the cockpit fleet's matrix philosophy without touching the underlying e2e harness.
183+
- Expected wall-time: ~6 min → ~2-3 min.
184+
185+
## ⚠️ Post-merge action item
186+
187+
GitHub branch protection rules currently require the singular check \`examples/chat — e2e\`. After this lands, that name no longer exists; the rule must be updated to require all four templated names:
188+
189+
- \`examples/chat — e2e (1/4)\`
190+
- \`examples/chat — e2e (2/4)\`
191+
- \`examples/chat — e2e (3/4)\`
192+
- \`examples/chat — e2e (4/4)\`
193+
194+
Until that update, future PR gates may fail to evaluate. **Repo admin should update branch protection immediately after merge.**
195+
196+
## Test plan
197+
198+
- [ ] All four \`examples/chat — e2e (N/4)\` matrix entries report success on this PR
199+
- [ ] Per-shard wall-time is ~2-3 min (vs. ~6 min single-job baseline)
200+
- [ ] Aggregate test count across shards is 31 (no test runs twice, no test is dropped)
201+
- [ ] No flake surfaced by the changed relative test order
202+
203+
Spec: \`docs/superpowers/specs/2026-05-21-examples-chat-e2e-shard-design.md\`
204+
Plan: \`docs/superpowers/plans/2026-05-21-examples-chat-e2e-shard.md\`
205+
206+
🤖 Generated with [Claude Code](https://claude.com/claude-code)
207+
EOF
208+
)"
209+
```
210+
211+
Expected: a URL to the new PR is printed.
212+
213+
- [ ] **Step 3: Wait for CI to start, then list the new check names**
214+
215+
Run (a minute or two after the push):
216+
217+
```bash
218+
gh pr checks $(gh pr view --json number --jq .number)
219+
```
220+
221+
Expected: 4 lines containing `examples/chat — e2e (1/4)` … `(4/4)`. If only 1 line for the old name appears, the matrix was not picked up — recheck Step 2 of Task 1.
222+
223+
- [ ] **Step 4: After CI completes, verify per-shard outcomes**
224+
225+
Run:
226+
227+
```bash
228+
PR=$(gh pr view --json number --jq .number)
229+
RUN=$(gh run list --branch claude/chat-e2e-shard --workflow=ci.yml --limit 1 --json databaseId --jq '.[0].databaseId')
230+
gh run view $RUN --json jobs --jq '.jobs[] | select(.name | startswith("examples/chat — e2e")) | {name, conclusion, startedAt, completedAt}'
231+
```
232+
233+
Expected: 4 entries, each with `conclusion: "success"`. Note `startedAt`/`completedAt` to compute per-shard duration; the slowest shard is the new wall-time of this job.
234+
235+
- [ ] **Step 5: Confirm aggregate test count from one shard's log (sanity check)**
236+
237+
Run:
238+
239+
```bash
240+
JOB_ID=$(gh run view $RUN --json jobs --jq '.jobs[] | select(.name=="examples/chat — e2e (1/4)") | .databaseId')
241+
gh run view --job=$JOB_ID --log 2>&1 | grep -E "[0-9]+ passed" | tail -3
242+
```
243+
244+
Expected: a line like `N passed (Mm Ss)` where N is roughly 31/4 ≈ 7-8 tests. Repeat for the other shards mentally; rough sum should be ≥31 (Playwright may report retries as additional "passed" entries).
245+
246+
- [ ] **Step 6: Hand off to user for merge decision**
247+
248+
The plan ends here. The user decides when to merge (and remembers to update branch protection — see PR body).
249+
250+
---
251+
252+
## Verification checklist (entire plan)
253+
254+
After all tasks: verify against spec at `docs/superpowers/specs/2026-05-21-examples-chat-e2e-shard-design.md`:
255+
256+
- ✅ Matrix is 4-way with numeric shard values
257+
- ✅ `fail-fast: false`
258+
- ✅ Job display name templated with shard index
259+
- ✅ `--shard=N/4` passed through Nx `--` separator
260+
- ✅ `--skip-nx-cache` preserved
261+
- ✅ Trace artifact name templated with shard index
262+
- ✅ No application code, test code, or harness change
263+
- ✅ `playwright.config.ts` unchanged
264+
- ✅ Downstream `needs:` references untouched
265+
- ✅ PR body flags the required-checks update for repo admin
266+
267+
If any item is unchecked, return to the task that owns it before requesting review.

0 commit comments

Comments
 (0)