Fix backport AI permission, surface infra failures, and allow manual dispatch#1943
Merged
TooTallNate merged 3 commits intomainfrom May 5, 2026
Merged
Conversation
…loud on AI infra errors Three related fixes triggered by the failed run on #1935: 1. Hoist the AI model name to a top-level `AI_MODEL` env var (`anthropic/claude-opus-4.7`); both `opencode run` invocations now interpolate `vercel/${AI_MODEL}` so the model is specified in exactly one place. 2. Switch `OPENCODE_PERMISSION` from the bare-string shortcut `"allow"` to the explicit object form `{"*":"allow","external_directory":"allow"}`. The shortcut was observed not to override `external_directory` (which defaults to "ask" and auto-rejects in non-interactive `opencode run`), causing the conflict-resolution AI to fail when reading scratch files it created under `/tmp/`. 3. The `Resolve conflicts with opencode` step no longer uses `continue-on-error`, and now distinguishes two outcomes via an AI- written outcome file (`.backport-conflict-outcome.json`): - `{"status":"resolved"}` — the legitimate clean path; cherry-pick continues and the backport PR is opened. - `{"status":"unresolved", ...}` — the legitimate "AI couldn't do it, hand off to a human" path; `resolved=false` is set and the conflict-failure comment is posted on the source PR. - Anything else (missing file, malformed JSON, unknown status) is treated as an opencode/AI Gateway infra failure: the step exits non-zero, the workflow fails red, and the misleading "couldn't resolve" comment is suppressed. The prompt + scratch files are also moved into the workspace so opencode never needs `external_directory` access anyway.
|
Contributor
Contributor
📊 Benchmark Results
workflow with no steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) workflow with 1 step💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) workflow with 10 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) workflow with 25 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) workflow with 50 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) Promise.all with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) Promise.all with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) Promise.all with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) Promise.race with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) Promise.race with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) Promise.race with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) workflow with 10 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) workflow with 25 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) workflow with 50 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) workflow with 10 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) workflow with 25 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) workflow with 50 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Nitro Stream Benchmarks (includes TTFB metrics)workflow with stream💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) stream pipeline with 5 transform steps (1MB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) 10 parallel streams (1MB each)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) fan-out fan-in 10 streams (1MB each)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) SummaryFastest Framework by WorldWinner determined by most benchmark wins
Fastest World by FrameworkWinner determined by most benchmark wins
Column Definitions
Worlds:
❌ Some benchmark jobs failed:
Check the workflow run for details. |
Contributor
🧪 E2E Test Results❌ Some tests failed Summary
❌ Failed Tests▲ Vercel Production (4 failed)nextjs-turbopack (1 failed):
nitro (1 failed):
sveltekit (2 failed):
Details by Category❌ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
❌ Some E2E test jobs failed:
Check the workflow run for details. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens the backport.yml GitHub Actions workflow so AI-driven decisioning and conflict resolution are more reliable and don’t silently succeed when the underlying opencode/AI infrastructure fails.
Changes:
- Hoists the AI model into a single top-level
AI_MODELenv var and uses it consistently acrossopencode runcalls. - Fixes
OPENCODE_PERMISSIONto an explicit JSON object form to ensureexternal_directoryis actually allowed in headless runs. - Removes
continue-on-errorfrom the conflict-resolution step and introduces an AI-written outcome file to distinguish “legitimate unresolved conflicts” from infra/tool failures (and gates the “manual resolution needed” comment accordingly).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
… PR body
Add a `workflow_dispatch` trigger to the backport workflow with two
optional inputs:
- `ref` — commit SHA on `main` to back-port (defaults to `main` HEAD)
- `model` — overrides the default AI model used by opencode for the
decision and conflict-resolution steps (defaults to the workflow's
hardcoded `AI_MODEL`)
The top-level `AI_MODEL` env var now uses
`${{ inputs.model || 'anthropic/claude-opus-4.7' }}` so manual runs
pick up the override without changing anything else.
Manual dispatch (like the `backport-stable` label) always forces a
backport regardless of any AI verdict — the operator's intent is
explicit by virtue of triggering the workflow. The PR body shows
"Triggered manually via `workflow_dispatch`." in that case.
The PR body's conflict-resolution attribution also now interpolates
`${AI_MODEL}` (e.g. "opencode with `anthropic/claude-opus-4.7`")
instead of hardcoding "Claude Opus" so the text stays accurate if the
default model is later changed.
TooTallNate
changed the title
karthikscale3
approved these changes
May 5, 2026
The previous `Resolve conflicts with opencode` sanity check used `git diff --diff-filter=U` to detect unresolved cherry-pick conflicts, which only catches unmerged index entries. That misses the case where the AI runs `git add` on a file that still has `<<<<<<<` / `=======` / `>>>>>>>` markers in its content — git happily stages the broken file as a normal modification. Add a second check using `git diff --check --cached`, which emits `leftover conflict marker` lines when any staged content still has the standard markers. Grep specifically for that phrase so unrelated whitespace warnings don't trip the check. Also update the inline comment to accurately describe what each check covers (per Copilot's review on #1943).
Contributor
|
No backport to This commit only modifies the backport workflow itself ( To override, add the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four related improvements to the backport workflow, triggered by the failed run on #1935 (run 25397238001) that ended green despite the conflict-resolution AI step actually failing:
1. Fix
OPENCODE_PERMISSIONforexternal_directorySwitch from the bare-string shortcut
'"allow"'to the explicit object form'{"*":"allow","external_directory":"allow"}'. The shortcut was observed not to overrideexternal_directory(which defaults to"ask"and auto-rejects in non-interactiveopencode run), causing:…when the conflict-resolution AI tried to read a scratch file it had just created under
/tmp/. The conflict-resolution prompt + scratch files are also moved into the workspace (.backport-conflict-prompt.txtetc.) so opencode never needsexternal_directoryaccess at all — same pattern the decision step already uses.2. Fail loud on conflict-resolution infra failures
Previously the
Resolve conflicts with opencodestep hadcontinue-on-error: true, so an opencode infra failure (auth error, rejected tool call, crash) would silently mark the step as success, leaveresolvedunset, and the workflow would post a misleading "AI couldn't resolve conflicts" comment on the source PR — making it look like a legitimate "needs human resolution" outcome.Now the step distinguishes three cases via an AI-written outcome file (
.backport-conflict-outcome.json):{"status":"resolved"}{"status":"unresolved","reason":"..."}resolved=falseis set; the conflict-failure comment is posted on the source PR (the legitimate "needs human help" path).continue-on-error), workflow fails red, and the misleading "couldn't resolve" comment is suppressed (Comment on conflict failureis now gated onresolved == 'false'instead of!= 'true').The decision step's existing fail-loud behavior already handled this for the AI-decision path; this brings the conflict-resolution step in line.
3. Hoist
AI_MODELto a single env varAdded a top-level
env: { AI_MODEL: ... }so the model is specified in exactly one place. Bothopencode runinvocations now use--model "vercel/${AI_MODEL}"(thevercel/provider prefix stays at the use site). The PR body's conflict-resolution attribution also interpolates${AI_MODEL}(e.g. "opencode withanthropic/claude-opus-4.7") instead of hardcoding "Claude Opus", so the text stays accurate if we change models later.4. Allow manual
workflow_dispatchwithref+modelinputsAdded a
workflow_dispatchtrigger with two optional inputs:ref— commit SHA onmainto back-port (defaults tomainHEAD; resolved viagh api repos/.../commits/<ref>)model— overrides the AI model for that one run (e.g. quick A/B against a different model)The top-level
AI_MODELis now${{ inputs.model || 'anthropic/claude-opus-4.7' }}, keeping a single source of truth. Manual dispatch (like thebackport-stablelabel) always forces a backport regardless of any AI verdict — operator intent is explicit by virtue of triggering the workflow. The PR body shows "Triggered manually viaworkflow_dispatch." in that case.AGENTS.mdis updated to document the new manual-dispatch trigger and its inputs.