Skip to content

ENG-3651/ENG-3652 Add Prime release e2e checks#644

Open
d42me wants to merge 2 commits into
mainfrom
feature/eng-3651-eng-3652-e2e-release-tests
Open

ENG-3651/ENG-3652 Add Prime release e2e checks#644
d42me wants to merge 2 commits into
mainfrom
feature/eng-3651-eng-3652-e2e-release-tests

Conversation

@d42me
Copy link
Copy Markdown
Contributor

@d42me d42me commented May 13, 2026

Adds a sandbox-backed release e2e workflow for prime env push/install, local eval uploads, and hosted eval submissions before release merges.


Note

Medium Risk
Adds a new GitHub Action and companion script that uses real API credentials to create sandboxes, push environments, and run/submit hosted evals, so failures could consume resources or affect remote state. Logic is isolated to CI tooling, but it exercises multiple external services and cleanup paths.

Overview
Introduces a new Release E2E Tests GitHub workflow (.github/workflows/release-e2e.yml) that runs on release/* PRs (or manually) and executes release smoke tests inside a Prime sandbox using repo-provided SDKs and configured secrets/vars.

Adds packages/prime/scripts/release_e2e.py, which archives the current checkout (skipping common build artifacts, symlinks, and likely-secret files), provisions a sandbox, installs the candidate CLI from the uploaded source, then validates core flows: prime env push/info/install, local eval runs + eval push, and optional hosted eval submission with configurable wait/cancel behavior and best-effort cleanup.

Includes tests (packages/prime/tests/test_release_e2e_script.py) to ensure archive filtering, remote-script compilation/cleanup behavior, and workflow input handling are correct (and that any old workflow name/path isn’t present).

Reviewed by Cursor Bugbot for commit e8c98d7. Bugbot is set up for automated code reviews on this repo. Configure here.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6d935eafc3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/release-e2e.yml
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 6d935ea. Configure here.

Comment thread .github/workflows/prime-release-e2e.yml Outdated
Comment thread packages/prime/scripts/release_e2e.py Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant