Skip to content

Add agentic product testing: agents verify the product as real users#2

Merged
0xhackerfren merged 1 commit into
mainfrom
product-testing
May 14, 2026
Merged

Add agentic product testing: agents verify the product as real users#2
0xhackerfren merged 1 commit into
mainfrom
product-testing

Conversation

@0xhackerfren
Copy link
Copy Markdown
Owner

Summary

  • Adds a new verification layer (Phase 5b) where AI agents operate the assembled application as real users with defined profiles and goal-oriented objectives
  • Captures UX quality metrics (click count, navigation depth, backtracking, error encounters, confusion index) with quantitative UX budgets as quality gates
  • This is a uniquely agentic capability: no previous testing methodology could instruct a test to 'figure out how to accomplish this goal' and measure whether the experience was efficient and discoverable

New files

  • reference/product-testing.md: Full reference covering user profiles, test objectives, UX budgets, experience signals, pipeline integration, agent behavior, metrics, and feedback to specification
  • harness/templates/product-test-template.md: Template for defining product test scenarios with objectives, budgets, and spec traceability
  • harness/templates/user-profile-template.md: Template for defining simulated user personas with technical literacy, domain knowledge, constraints, and behavioral tendencies

Updated files

  • MANIFESTO.md: Product testing added to Section VI (The Machine) -- the assembled product verified by agents acting as users
  • harness/PLAYBOOK.md: Phase 5b (Product Testing) between Review and Deployment, with inputs, outputs, exit criteria, human/agent responsibilities, quick reference row
  • harness/templates/test-plan-template.md: Product test section for defining objectives alongside module-level molds
  • reference/workflow.md: Product testing as step 7 in the pipeline, flow diagram updated, kanban stations table updated, companion list updated
  • reference/glossary.md: Four new terms (product test, test objective, user profile, UX budget)
  • harness/.cursor/rules/codex-automata.mdc: Product testing section and terminology
  • README.md: Product testing section
  • CHANGELOG.md: Unreleased entries
  • ROADMAP.md: Current section

Test plan

  • Verify reference/product-testing.md cross-references resolve
  • Verify templates are self-contained (GitHub URLs for references outside harness)
  • Verify glossary terms are in alphabetical order
  • Verify workflow.md flow diagram includes product testing
  • Run markdownlint CI

Adds a new verification layer where AI agents operate the assembled application as real users. Each agent receives a user profile (technical literacy, domain knowledge, constraints) and a goal-oriented objective, then navigates the product through its UI. Captures UX quality metrics: click count, navigation depth, backtracking, error encounters, confusion index. UX budgets set quantitative thresholds as quality gates. This is a uniquely agentic capability that no previous testing methodology could provide.
@0xhackerfren 0xhackerfren merged commit ff37484 into main May 14, 2026
@0xhackerfren 0xhackerfren deleted the product-testing branch May 14, 2026 17:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant