Skip to content

Add CircleCI job for maestro E2E tests#1699

Merged
ajpallares merged 25 commits intomainfrom
add-maestro-e2e-test-ci-job
Apr 15, 2026
Merged

Add CircleCI job for maestro E2E tests#1699
ajpallares merged 25 commits intomainfrom
add-maestro-e2e-test-ci-job

Conversation

@ajpallares
Copy link
Copy Markdown
Member

@ajpallares ajpallares commented Mar 25, 2026

Summary

  • Adds two separate CircleCI workflows triggered by the maestro_e2e_tests schedule:
    • maestro-e2e-tests-ios: runs on macOS executor, builds the test app for iOS simulator via flutter build ios --debug --simulator, runs Maestro tests
    • maestro-e2e-tests-android: runs on android:2024.11.1 machine image, creates an Android emulator, builds a debug APK via flutter build apk --debug, runs Maestro tests
  • Adds Fastlane lanes with upfront API key validation (UI.user_error! if env var is missing):
    • run_maestro_e2e_tests_ios: replaces API key placeholder, builds with Flutter, installs on simulator, runs Maestro
    • build_maestro_app_android / run_maestro_e2e_tests_android: replaces API key placeholder, builds APK, installs via adb, runs Maestro
  • Both lanes use $RC_E2E_TEST_API_KEY_PRODUCTION_TEST_STORE from the e2e-tests CircleCI context
  • Test results stored as JUnit artifacts

Depends on #1698


Note

Medium Risk
CI pipeline changes add new scheduled iOS/Android emulator runs and Fastlane scripting that mutates the E2E test app at build time; main risk is increased CI flakiness or misconfigured secrets causing failures rather than product impact.

Overview
Adds Maestro end-to-end testing to CircleCI by introducing new run-maestro-e2e-tests-ios and run-maestro-e2e-tests-android jobs that build and run a dedicated Flutter test app on an iOS simulator and an Android emulator, then upload JUnit results as artifacts.

Wires these jobs into the main test workflow (using the e2e-tests context) and also adds dedicated scheduled workflows (maestro_e2e_tests) to run iOS/Android E2E suites independently.

Extends the Fastfile with lanes to build/install the Maestro test app and execute maestro test, including validation of RC_E2E_TEST_API_KEY_PRODUCTION_TEST_STORE and replacing an API-key placeholder before running the suite.

Reviewed by Cursor Bugbot for commit b01d19b. Bugbot is set up for automated code reviews on this repo. Configure here.

@ajpallares ajpallares force-pushed the add-maestro-e2e-test branch from 96cad0e to 5f4fc0e Compare March 25, 2026 16:09
@ajpallares ajpallares force-pushed the add-maestro-e2e-test-ci-job branch from 8f8eb7b to effa822 Compare March 25, 2026 16:10
@ajpallares ajpallares force-pushed the add-maestro-e2e-test branch from 5f4fc0e to 979b0a5 Compare March 25, 2026 16:20
@ajpallares ajpallares force-pushed the add-maestro-e2e-test-ci-job branch from effa822 to ab777d6 Compare March 25, 2026 16:20
@ajpallares ajpallares force-pushed the add-maestro-e2e-test branch from 979b0a5 to 100d385 Compare March 25, 2026 16:24
@ajpallares ajpallares force-pushed the add-maestro-e2e-test-ci-job branch 2 times, most recently from 4534287 to 43395b2 Compare March 25, 2026 16:45
@ajpallares ajpallares force-pushed the add-maestro-e2e-test branch from 991867f to a5cac25 Compare March 25, 2026 17:09
@ajpallares ajpallares force-pushed the add-maestro-e2e-test-ci-job branch 3 times, most recently from b2dda81 to 5ad518b Compare March 30, 2026 08:33
@ajpallares ajpallares force-pushed the add-maestro-e2e-test branch from e776f81 to 89bd52c Compare March 30, 2026 11:16
@ajpallares ajpallares force-pushed the add-maestro-e2e-test-ci-job branch 3 times, most recently from cfe5590 to da877f5 Compare March 30, 2026 11:45
@ajpallares ajpallares force-pushed the add-maestro-e2e-test branch from 1cf8fed to b266f9e Compare March 30, 2026 14:41
@ajpallares ajpallares force-pushed the add-maestro-e2e-test-ci-job branch from da877f5 to 1fcdee4 Compare March 30, 2026 14:41
@ajpallares ajpallares force-pushed the add-maestro-e2e-test branch from b266f9e to 82563b8 Compare March 30, 2026 15:03
@ajpallares ajpallares force-pushed the add-maestro-e2e-test-ci-job branch from 1fcdee4 to 0265781 Compare March 30, 2026 15:03
@ajpallares ajpallares force-pushed the add-maestro-e2e-test branch from 82563b8 to 067a74c Compare March 30, 2026 15:16
@ajpallares ajpallares force-pushed the add-maestro-e2e-test-ci-job branch from 0265781 to 073a93f Compare March 30, 2026 15:16
@ajpallares ajpallares force-pushed the add-maestro-e2e-test branch from 067a74c to 6893fcf Compare March 30, 2026 15:30
@ajpallares ajpallares force-pushed the add-maestro-e2e-test-ci-job branch from 073a93f to 65ab3dc Compare March 30, 2026 15:30
@ajpallares ajpallares force-pushed the add-maestro-e2e-test branch from 6893fcf to c01f963 Compare March 30, 2026 17:06
@ajpallares ajpallares force-pushed the add-maestro-e2e-test-ci-job branch from 1f6ac86 to e6fe691 Compare March 30, 2026 17:06
@ajpallares ajpallares force-pushed the add-maestro-e2e-test branch from c01f963 to 691052f Compare March 30, 2026 17:21
@ajpallares ajpallares force-pushed the add-maestro-e2e-test-ci-job branch from da16a41 to 5f321fc Compare March 30, 2026 17:21
Made-with: Cursor
Replace manual flutter precache + pod install + xcodebuild with
the idiomatic `flutter build ios --debug --simulator`, which handles
all intermediate steps automatically.

Made-with: Cursor
Fail fast with a clear error if RC_E2E_TEST_API_KEY_PRODUCTION_TEST_STORE
is not set, instead of silently replacing the placeholder with an empty
string and crashing at runtime.

Made-with: Cursor
@ajpallares ajpallares force-pushed the add-maestro-e2e-test-ci-job branch from 028102c to 0fe57e2 Compare April 8, 2026 12:42
ajpallares and others added 14 commits April 15, 2026 10:59
The test app no longer uses CocoaPods (removed in PR1), so there's
no need to install it for the Maestro E2E tests.

Made-with: Cursor
- Remove Podfile and Podfile.lock (project already uses SPM via
  FlutterGeneratedPluginSwiftPackage; zero CocoaPods refs in pbxproj)
- Remove Pods #include? from Debug/Release xcconfig
- Add enable-swift-package-manager: true to pubspec.yaml
- Extract test case definitions to test_cases.dart for extensibility
  (single source of truth for title, flowKey, and screen widget)
- Refactor TestCasesScreen to build from the shared list

Made-with: Cursor
Avoids eagerly constructing all screen widgets upfront — the widget
is only created when the user navigates to it.

Made-with: Cursor
Only e2e-tests context is needed, matching purchases-ios.

Made-with: Cursor
Fail fast if the placeholder was not replaced, matching KMP.

Made-with: Cursor
@ajpallares ajpallares marked this pull request as ready for review April 15, 2026 10:55
@ajpallares ajpallares requested a review from a team as a code owner April 15, 2026 10:55
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit b01d19b. Configure here.

Comment thread .circleci/config.yml
- e2e-tests
- run-maestro-e2e-tests-android:
context:
- e2e-tests
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

E2E tests unintentionally run on every push/PR

Medium Severity

The run-maestro-e2e-tests-ios and run-maestro-e2e-tests-android jobs are added to the test workflow, which fires on every non-scheduled pipeline trigger (i.e., every push and PR). This means these expensive E2E tests — requiring a macOS m4pro.medium executor and an Android xlarge machine executor — will run on every commit, not just on the maestro_e2e_tests schedule as described in the PR. The dedicated scheduled workflows at lines 666–682 already handle the scheduled runs, making this test-workflow inclusion likely unintentional and costly.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit b01d19b. Configure here.

ajpallares added a commit that referenced this pull request Apr 15, 2026
## Summary

Adds a Maestro E2E test app under `e2e-tests/MaestroTestApp/` — a
minimal Flutter app with two screens used by automated Maestro flows to
verify the RevenueCat purchase integration.

- Two screens: "Test Cases" list and "Purchase through paywall"
(presents a RevenueCat V2 paywall and displays entitlement status)
- Bundle ID `com.revenuecat.automatedsdktests`
- API key placeholder (`MAESTRO_TESTS_REVENUECAT_API_KEY`) replaced at
CI time via `sed`
- Local SDK resolution via `path:` references + `dependency_overrides`
(same mechanism as `revenuecat_examples/purchase_tester`)
- iOS plugins resolved via Swift Package Manager
(`FlutterGeneratedPluginSwiftPackage`); CocoaPods only carries the
Flutter engine
- Errors from `getCustomerInfo` and `presentPaywall` are surfaced in the
UI for debugging failed Maestro flows

Counterpart PRs:
[react-native-purchases#1635](RevenueCat/react-native-purchases#1635),
[purchases-capacitor#699](RevenueCat/purchases-capacitor#699),
[cordova-plugin-purchases#857](RevenueCat/cordova-plugin-purchases#857),
[purchases-unity#836](RevenueCat/purchases-unity#836),
[purchases-kmp#708](RevenueCat/purchases-kmp#708)

### Follow-up PRs (stacked)
- [#1698](#1698) —
Maestro test flows (YAML)
- [#1699](#1699) —
CircleCI jobs to run the tests
- [#1714](#1714) —
Launch argument routing to skip test list

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Low Risk**
> Adds a new, isolated test-only Flutter project with standard
iOS/Android scaffolding and a simple paywall purchase flow; it does not
modify the SDK runtime code paths.
> 
> **Overview**
> Introduces a new Flutter app in `e2e-tests/MaestroTestApp` intended
for Maestro E2E automation, including a test-case list screen and a
purchase flow that calls `RevenueCatUI.presentPaywall()` and displays
active `pro` entitlement state.
> 
> Adds full iOS/Android project scaffolding (manifests, Gradle/Xcode
configs, app IDs) plus local `path:` dependencies on
`purchases_flutter`/`purchases_ui_flutter`, and documents CI-time API
key injection via the `MAESTRO_TESTS_REVENUECAT_API_KEY` placeholder.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
cb1f2f4. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
Base automatically changed from add-maestro-e2e-test to main April 15, 2026 11:03
@ajpallares ajpallares merged commit 12069ef into main Apr 15, 2026
17 checks passed
@ajpallares ajpallares deleted the add-maestro-e2e-test-ci-job branch April 15, 2026 11:03
RCGitBot added a commit that referenced this pull request Apr 17, 2026
**This is an automatic release.**

## RevenueCat SDK
### 🐞 Bugfixes
* [EXTERNAL] Fix Wasm incompatibility in web error processing (#1684)
contributed by @brunovsiqueira (#1722) via Toni Rico (@tonidero)
### 📦 Dependency Updates
* [AUTOMATIC BUMP] Updates purchases-hybrid-common to 18.1.0 (#1726) via
RevenueCat Git Bot (@RCGitBot)
* [Android
10.2.0](https://github.com/RevenueCat/purchases-android/releases/tag/10.2.0)
* [iOS
5.68.0](https://github.com/RevenueCat/purchases-ios/releases/tag/5.68.0)

### 🔄 Other Changes
* Add retry mechanism to Maestro e2e tests (#1723) via Antonio Pallares
(@ajpallares)
* Bump fastlane-plugin-revenuecat_internal from `20911d1` to `a1eed48`
(#1725) via dependabot[bot] (@dependabot[bot])
* Skip test cases list in maestro tests using launch arguments (#1714)
via Antonio Pallares (@ajpallares)
* Add CircleCI job for maestro E2E tests (#1699) via Antonio Pallares
(@ajpallares)
* Add maestro E2E test for purchase through paywall (#1698) via Antonio
Pallares (@ajpallares)
* Add maestro E2E test app (#1654) via Antonio Pallares (@ajpallares)

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Low Risk**
> Low risk release bookkeeping: primarily version bumps and
changelog/version-matrix updates, with no functional logic changes in
the plugin code beyond reporting the new version string.
> 
> **Overview**
> Updates the SDKs to **v`10.0.1`** by bumping version identifiers
across `pubspec.yaml`, Android Gradle configs, and iOS/macOS podspecs
(including the runtime-reported plugin version strings).
> 
> Refreshes release documentation (`CHANGELOG.md`,
`CHANGELOG-LATEST.md`, `VERSIONS.md`) to include the `10.0.1` notes
(Wasm web error-processing fix) and the dependency bump to
`purchases-hybrid-common` `18.1.0` (Android `10.2.0`, iOS `5.68.0`).
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
ff0bb5e. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants