🤖 QA Automation Agent

An AI agent that reads your requirements, tests your staging app, and alerts your team — automatically.

Stop manually testing before every deploy. This agent reads your Notion docs, drives a real browser through your user flows, spots bugs using Claude, and pings your Slack with a one-click approve button that ships to prod.

View Demo · Quick Start · Architecture · Contributing

Live Deployment

API base URL: https://<your-domain>
Health check: https://<your-domain>/health

What it does

Stage 1 — Pre-deploy QA (runs on every PR merge to staging)

Reads your feature requirements from a Notion page
Claude extracts structured acceptance criteria from your prose
Claude generates a human-readable browser action plan
Playwright executes every step, taking screenshots along the way
Claude evaluates the results against your requirements
If a bug is found → Slack alert with the action plan, screenshot, severity label, and one-click Approve & Deploy / Dismiss buttons

Stage 2 — Post-deploy validation (runs after prod deploy)

Pulls the last hour of Amplitude event streams from production
Claude reads the session sequences against the same Notion requirements
If real users are hitting unexpected paths → alerts the developer who pushed

Notion ──► requirementsParser ──► planGenerator ──► Playwright ──► resultEvaluator ──► Slack
                                                                                          │
                                                                              Approve ────┤
                                                                                          ▼
                                                                               GitHub Actions
                                                                                          │
                                                                                          ▼
                                                                    Amplitude ──► Claude ──► Slack

Demo

Stage 1 fires when a PR is merged to staging. The agent navigates the staging URL, fills the form, clicks submit, and catches the bug:

🔴 Bug Detected — demo-contact-form
Severity: critical  |  Confidence: high

What went wrong:
After clicking Submit, the page redirected to /error instead of showing
the success message "Thank you! We'll be in touch." as required.

Action plan Claude executed:
1. Navigate to https://staging.example.com/contact → Contact form visible
2. Fill "Name" field with "Test User" → Name field populated
3. Fill "Email" field with "test@example.com" → Email field populated
4. Fill "Message" field with "Hello" → Message field populated
5. Click "Submit" button → Success message appears

Triggered by: [View Notion requirement →]

[ ✅ Approve & Deploy to Prod ]   [ ❌ Dismiss ]

Architecture

qa-agent/
├── agent/
│   ├── claudeClient.js          # Shared Anthropic API wrapper
│   ├── requirementsParser.js    # Claude Call 1 — Notion prose → structured JSON
│   ├── planGenerator.js         # Claude Call 2 — requirements → action plan
│   ├── resultEvaluator.js       # Claude Call 3 — screenshots + page state → bug report
│   ├── stage1.js                # Stage 1 orchestrator
│   └── stage2.js                # Stage 2 orchestrator
├── browser/
│   └── playwrightRunner.js      # Headless browser — executes action plans
├── integrations/
│   ├── notionClient.js          # Fetches page content
│   ├── amplitudeClient.js       # Pulls session event streams
│   ├── slackClient.js           # Sends alerts + interactive buttons
│   └── githubClient.js          # Dispatches workflow runs
├── db/
│   ├── schema.sql               # feature_runs table
│   └── queries.js               # Raw pg queries
├── server/
│   └── index.js                 # Express: /slack/interact + /run-check
├── config/
│   └── team.config.json         # GitHub username → Slack user ID map
├── scripts/
│   └── seed-demo.js             # Seeds a demo run + instructions to add a bug
└── .github/workflows/
    ├── stage1-qa-check.yml      # Triggers on PR merge to staging
    └── stage2-prod-verify.yml   # Triggers via workflow_dispatch after prod deploy

Three focused Claude calls, each independently logged to Postgres. If one fails, the others don't re-run. Every input and output is stored — full audit trail per run.

Quick Start

Prerequisites

Node.js 20+
A Railway account (free tier — for Postgres + Express hosting)
A Notion workspace with API access
An Amplitude account (free tier works)
A Slack workspace where you can create apps

1. Clone and install

git clone https://github.com/your-username/qa-agent.git
cd qa-agent
npm install
npx playwright install chromium

2. Set up environment variables

cp .env.example .env

Open .env and fill in every value. See the Environment Variables section for where to get each one.

3. Set up the database

In Railway: create a new Postgres service → copy the connection string into DATABASE_URL. Then:

psql $DATABASE_URL -f db/schema.sql

Expected output: CREATE TABLE

4. Configure your team

Edit config/team.config.json:

{
  "userMap": {
    "your-github-username": "YOUR_SLACK_USER_ID"
  }
}

Find your Slack user ID: Slack → your profile → three-dot menu → Copy member ID.

5. Create your Notion requirements page

Create a Notion page with this structure (prose is fine — no database needed):

Feature: Contact Form

Users should be able to submit a contact form with their name, email, and message.

Acceptance Criteria:
- The form has fields for Name, Email, and Message
- Clicking Submit sends the form
- A success message "Thank you! We'll be in touch." appears after submission
- The form fields are cleared after successful submission

Relevant URL: https://your-staging-url.com/contact

Copy the page ID from the Notion URL (the 32-character string after the last -). Add it to .env as NOTION_PAGE_ID.

6. Set up the Slack app

Go to api.slack.com/apps → Create New App → From scratch
Incoming Webhooks → Activate → Add to workspace → copy URL → SLACK_WEBHOOK_URL
Interactivity & Shortcuts → Enable → set Request URL to https://<your-domain>/slack/interact
OAuth & Permissions → Bot scopes: chat:write, files:write → Install to workspace → copy Bot Token → SLACK_BOT_TOKEN
Basic Information → copy Signing Secret → SLACK_SIGNING_SECRET

7. Run tests

Run these in order. Each group depends on the previous passing.

# Infrastructure
npm test tests/db.test.js
npm test tests/claudeClient.test.js

# Integrations
npm test tests/notionClient.test.js
npm test tests/amplitudeClient.test.js

# Agent modules
npm test tests/requirementsParser.test.js
npm test tests/planGenerator.test.js
npm test tests/playwrightRunner.test.js
npm test tests/resultEvaluator.test.js

# Slack + GitHub
npm test tests/slackClient.test.js      # check Slack channel for the message
npm test tests/getSlackUser.test.js
npm test tests/githubClient.test.js

# End-to-end
npm test tests/stage1.test.js           # full pipeline — Slack alert should fire
npm test tests/stage2.test.js           # requires stage1 run to exist first

8. Deploy the Express server

Push to GitHub → Railway → New Project → Deploy from GitHub repo → add all .env values as Railway environment variables. Railway auto-detects npm start.

Set your Slack app's Interactivity Request URL to https://<your-domain>/slack/interact.

Running the demo

Option A — Manual trigger (fastest for demos)

npm start
curl -X POST "http://localhost:3000/run-check?feature=<your-notion-page-id>"

Option B — Automated via PR

Open a PR against the staging branch. Paste your Notion page ID as the PR body. Merge it — Stage 1 fires automatically.

Introduce a deliberate bug for the demo

In your staging app, make the form submit handler redirect to /error instead of showing the success message. Stage 1 will catch it, fire the Slack alert, and show the Approve/Dismiss buttons.

Seed demo data

npm run seed

15-Minute Handover Checklist

Use this to onboard the QA agent into a new team/project quickly.

Fork or clone this repository into the target GitHub account.
Create and fill .env from .env.example.
Set up Postgres and run schema:
- psql $DATABASE_URL -f db/schema.sql
Configure Slack app:
- webhook URL
- bot token
- signing secret
- interactivity URL: https://<your-domain>/slack/interact
Configure Notion:
- create integration
- share requirements page with integration
- set NOTION_PAGE_ID
Configure Amplitude keys for prod session verification.
Configure GitHub token and repo envs (GITHUB_OWNER, GITHUB_REPO).
Deploy backend service (Railway or equivalent persistent Node host).
Set custom domain and verify health:
- https://<your-domain>/health
Add GitHub Actions secrets and enable workflows.
Update config/team.config.json:
- map GitHub usernames to Slack member IDs.
Run validation tests in order:
- npm test tests/db.test.js
- npm test tests/stage1.test.js
- npm test tests/stage2.test.js

After this checklist, teams can trigger Stage 1 from PR merges or manual API calls and use Slack Approve/Dismiss for controlled promotion to Stage 2 verification.

Environment Variables

Variable	Where to get it
`ANTHROPIC_API_KEY`	console.anthropic.com → API Keys
`NOTION_API_KEY`	notion.so/my-integrations → New integration
`NOTION_PAGE_ID`	From your Notion page URL
`AMPLITUDE_API_KEY`	Amplitude → Settings → Projects → API Key
`AMPLITUDE_SECRET_KEY`	Amplitude → Settings → Projects → Secret Key
`AMPLITUDE_EXPORT_DELAY_HOURS`	Optional. Defaults to `4` (handles Export API lag)
`AMPLITUDE_EXPORT_WINDOW_HOURS`	Optional. Defaults to `24` (query window size in hours)
`SLACK_WEBHOOK_URL`	Slack app → Incoming Webhooks
`SLACK_SIGNING_SECRET`	Slack app → Basic Information
`SLACK_BOT_TOKEN`	Slack app → OAuth & Permissions
`AI_PROVIDER`	`anthropic`, `gemini`, or `openai`
`ANTHROPIC_MODEL`	Optional Anthropic model override
`GEMINI_API_KEY`	Required only if `AI_PROVIDER=gemini`
`GEMINI_MODEL`	Optional Gemini model override
`OPENAI_API_KEY`	Required only if `AI_PROVIDER=openai`
`OPENAI_MODEL`	Optional OpenAI model override
`OPENAI_BASE_URL`	Optional OpenAI-compatible base URL (for example, ChatMock)
`GITHUB_TOKEN`	GitHub → Settings → Developer Settings → Fine-grained tokens (needs `Actions: write`)
`GITHUB_OWNER`	Your GitHub username
`GITHUB_REPO`	Your repo name
`DATABASE_URL`	Railway Postgres → connection string
`STAGING_BASE_URL`	e.g. `https://your-app.vercel.app`
`PORT`	Default `3000`

Add all of these as GitHub Actions secrets too (Settings → Secrets → Actions). Use GH_TOKEN for your personal access token — GITHUB_TOKEN is reserved by GitHub Actions.

How Claude is used

Three separate, stateless API calls — each with a focused system prompt:

Call	Input	Output
`requirementsParser`	Raw Notion page content	`[{ requirement, acceptanceCriteria, relevantUrl }]`
`planGenerator`	Structured requirements	`[{ stepNumber, action, expectedOutcome }]`
`resultEvaluator`	Action plan + page states	`{ passed, severity, confidence, findings[], summary }`

Each call's input and output is logged to Postgres independently. If evaluation fails, you don't re-run the browser session. Every run is a full audit trail.

Default provider: anthropic (claude-sonnet-4-6). You can switch to Gemini/OpenAI via env.

Contributing

Contributions are welcome. A few areas where help would be great:

Better Playwright action parsing — the current step executor uses regex heuristics. A proper NLP-to-selector mapper would make it more reliable across diverse UIs
Screenshot diffing — attach screenshots to the Slack alert as image attachments
Multi-page requirement support — currently one Notion page per feature run; supporting linked pages would help larger teams
Structured Notion DB support — the requirements fetcher is abstracted behind getPageContent() — a DB adapter would be a clean swap

To contribute:

git checkout -b feat/your-feature
# make changes
npm test
git commit -m "feat: description"
# open a PR

Please run the full test suite before opening a PR.

License

MIT — use it, fork it, build on it.

Built with Claude API · Playwright · Notion API · Amplitude · Slack API

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

🤖 QA Automation Agent

Live Deployment

What it does

Demo

Architecture

Quick Start

Prerequisites

1. Clone and install

2. Set up environment variables

3. Set up the database

4. Configure your team

5. Create your Notion requirements page

6. Set up the Slack app

7. Run tests

8. Deploy the Express server

Running the demo

15-Minute Handover Checklist

Environment Variables

How Claude is used

Contributing

License

About

Uh oh!

Releases

Packages

Uh oh!

Contributors

Uh oh!

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
.github/workflows		.github/workflows
agent		agent
browser		browser
config		config
db		db
integrations		integrations
scripts		scripts
server		server
tests		tests
.env.example		.env.example
.gitignore		.gitignore
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
package-lock.json		package-lock.json
package.json		package.json

Folders and files

Latest commit

History

Repository files navigation

🤖 QA Automation Agent

Live Deployment

What it does

Demo

Architecture

Quick Start

Prerequisites

1. Clone and install

2. Set up environment variables

3. Set up the database

4. Configure your team

5. Create your Notion requirements page

6. Set up the Slack app

7. Run tests

8. Deploy the Express server

Running the demo

15-Minute Handover Checklist

Environment Variables

How Claude is used

Contributing

License

About

Topics

Resources

License

Code of conduct

Contributing

Security policy

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages