feat: add qa-changes plugin for automated PR QA validation

[xingyaoww]

Summary

Add a new qa-changes plugin that goes beyond code review by actually running the code to verify PR changes work as described. While the existing pr-review plugin reads diffs and posts inline code comments, this plugin sets up the environment, runs the test suite, exercises changed behavior, and posts a structured QA report.

Plugin Structure

skills/qa-changes/SKILL.md              # Generic QA methodology skill
plugins/qa-changes/
├── README.md                           # Plugin documentation
├── action.yml                          # Composite GitHub Action
├── scripts/
│   ├── agent_script.py                 # Main QA agent script
│   └── prompt.py                       # Prompt template
├── skills/
│   └── qa-changes -> ../../../skills/qa-changes
└── workflows/
    └── qa-changes-by-openhands.yml     # Example workflow

Five-Phase QA Methodology

The skill defines a generic, language-agnostic methodology:

1. Understand — Read the diff, classify changes (new feature, bug fix, refactor, config/docs)
2. Setup — Bootstrap the repo: install deps, build, establish test baseline
3. Test — Run the test suite, record pass/fail counts, detect regressions
4. Exercise — Go beyond tests: execute new features, reproduce fixed bugs, try edge cases
5. Report — Post structured PR comment with evidence and verdict (PASS / PASS WITH ISSUES / FAIL)

How It Differs from PR Review

Aspect	PR Review	QA Changes
Method	Reads the diff	Runs the code
Speed	2-3 minutes	5-15 minutes
Catches	Style, security, logic issues	Regressions, broken features, build failures
Output	Inline code comments	Structured QA report with evidence

Usage

- name: Run QA Changes
  uses: OpenHands/extensions/plugins/qa-changes@main
  with:
    llm-model: anthropic/claude-sonnet-4-5-20250929
    llm-api-key: ${{ secrets.LLM_API_KEY }}
    github-token: ${{ secrets.GITHUB_TOKEN }}

Triggers: qa-this label or openhands-agent reviewer request.

Design Decisions

• Generic skill: The SKILL.md is intentionally language/framework-agnostic. It teaches the agent how to think about QA, not specific commands. Project-specific details come from AGENTS.md or custom skills.
• Structured from pr-review: The agent_script.py and action.yml follow the same patterns as pr-review for consistency, but the prompt and skill are completely different.
• Security: Excludes FIRST_TIME_CONTRIBUTOR and NONE from automatic triggers since QA executes code.

Related

• Companion docs PR: OpenHands/docs (adds use-case page and SDK workflow guide)
• Inspired by Anthropic's harness design for long-running apps — the QA agent is essentially the "evaluator" in a generator-evaluator pattern.

This PR was created by an AI assistant (OpenHands).

@xingyaoww can click here to continue refining the PR