Skip to content

test: move replay coverage toward stable mobile app patterns #320

@thymikee

Description

@thymikee

Summary

The current replay coverage is useful, but too much of the core smoke/behavior validation still depends on Apple system apps and simulator-specific surfaces. That makes the suite less representative of real mobile app development patterns and more sensitive to OS churn and host-environment flakiness.

Proposed Improvements

  1. Move most smoke and interaction coverage off system apps and onto a controlled fixture app with stable accessibility identifiers.
  2. Keep system-app coverage only where it validates platform-specific behavior:
    • app launch / close
    • deep links
    • permissions / alerts
    • home / app switcher
    • screenshots / recordings
  3. Split replay coverage by intent:
    • smoke: fast, minimal confidence checks
    • behavior: selectors, fill, scroll, keyboard, lifecycle, alerts
    • benchmark/soak: longer end-to-end journeys
  4. Add flows that better match real mobile app usage:
    • cold start -> deep link -> navigation
    • text entry + keyboard show/dismiss
    • background -> foreground resume
    • modal/sheet open/close
    • permission prompt accept/dismiss/recovery
    • interrupted flow after home/app switcher
    • long-list rediscovery and scroll recovery
  5. Add one explicit host-focus regression for iOS simulator so we catch cases where Simulator steals macOS focus during automation.
  6. Prefer structural assertions over screenshot dependence in CI where possible.

Why This Matters

  • System Settings / Simulator UI changes are outside our control and make otherwise-valid tests flaky.
  • A fixture app gives us stable selectors and lets us model the patterns real mobile teams actually care about.
  • The iOS simulator focus issue is a host-OS-visible regression and should have dedicated end-to-end coverage.

Suggested Starting Point

  1. Build a small fixture/demo app with stable ids and common mobile UI patterns.
  2. Migrate smoke and behavior replays from Settings/system-app journeys to that fixture.
  3. Keep benchmarks as longer, more realistic user journeys.
  4. Add a focused iOS simulator “does not steal host focus” regression test.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions