ENG-3651/ENG-3652 Add Prime release e2e checks#644
Open
d42me wants to merge 2 commits into
Open
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6d935eafc3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 6d935ea. Configure here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Adds a sandbox-backed release e2e workflow for prime env push/install, local eval uploads, and hosted eval submissions before release merges.
Note
Medium Risk
Adds a new GitHub Action and companion script that uses real API credentials to create sandboxes, push environments, and run/submit hosted evals, so failures could consume resources or affect remote state. Logic is isolated to CI tooling, but it exercises multiple external services and cleanup paths.
Overview
Introduces a new Release E2E Tests GitHub workflow (
.github/workflows/release-e2e.yml) that runs onrelease/*PRs (or manually) and executes release smoke tests inside a Prime sandbox using repo-provided SDKs and configured secrets/vars.Adds
packages/prime/scripts/release_e2e.py, which archives the current checkout (skipping common build artifacts, symlinks, and likely-secret files), provisions a sandbox, installs the candidate CLI from the uploaded source, then validates core flows:prime env push/info/install, local eval runs +eval push, and optional hosted eval submission with configurable wait/cancel behavior and best-effort cleanup.Includes tests (
packages/prime/tests/test_release_e2e_script.py) to ensure archive filtering, remote-script compilation/cleanup behavior, and workflow input handling are correct (and that any old workflow name/path isn’t present).Reviewed by Cursor Bugbot for commit e8c98d7. Bugbot is set up for automated code reviews on this repo. Configure here.