Skip to content

Add public API commands#58

Open
ramilamparo wants to merge 29 commits intomainfrom
dev/ram/public-api
Open

Add public API commands#58
ramilamparo wants to merge 29 commits intomainfrom
dev/ram/public-api

Conversation

@ramilamparo
Copy link
Copy Markdown
Collaborator

@ramilamparo ramilamparo commented Mar 18, 2026

Summary

Adds a new qasphere api command that provides direct CLI access to the full QA Sphere public API. Users can now manage projects, test runs, test cases, results, and other resources programmatically without writing custom API integration code.

What's included

  • 16 resource subcommands: projects, runs, test-cases, results, folders, milestones, tags, requirements, shared-steps, shared-preconditions, custom-fields, audit-logs, settings, test-plans, files, users
  • JSON argument flexibility: Complex args accept inline JSON or @filename references
  • Zod validation with path-based errors: Detailed error messages like [0].tcaseIds: not allowed for "live" runs
  • Lazy env loading: QAS_URL/QAS_TOKEN loaded only when the API is called, so CLI validation errors surface first
  • Path parameter validation: Regex-based validation prevents injection via crafted identifiers
  • Agent Skills: New Skill.md file allows installing skills via npx skills add Hypersequent/qas-cli

Folder structure

src/api/                          # API client layer
├── index.ts                      # createApi() — composes fetch chain and assembles all sub-modules
├── schemas.ts                    # Shared types, RequestValidationError, validateRequest(), common Zod fields
├── utils.ts                      # Fetch decorators (withBaseUrl, withApiKey, withJson, withDevAuth)
└── <resource>.ts                 # One per resource — exports create<Resource>Api(fetcher) factory,
                                  #   defines Zod request schemas for early validation before API calls

src/commands/api/                 # CLI command layer
├── main.ts                       # Registers all resource subcommands
├── utils.ts                      # Shared helpers: apiHandler, parseAndValidateJsonArg,
│                                 #   handleValidationError, buildArgumentMap, printJson, etc.
└── <resource>/
    ├── command.ts                # Yargs command definitions and handler logic
    └── help.ts                   # Help text and descriptions

src/tests/api/                    # API command tests
├── test-helper.ts                # Shared infrastructure: useMockServer(), runCli(), test fixture
│                                 #   with project lifecycle (mock or live), helper creators
├── utils.spec.ts                 # Unit tests for API command utilities
└── <resource>/
    └── <action>.spec.ts          # One spec per action — mocked tests (MSW), validation error
                                  #   tests, and live tests (tagged, run against real instance)

Other changes

  • New API modules: audit-logs, custom-fields, milestones, requirements, results, settings, shared-preconditions, shared-steps, tags, test-plans, users, file — plus expanded runs, tcases, projects, folders
  • CLAUDE.md updated with full architecture docs for the API command
  • README.md updated with usage documentation
  • SKILL.md added for AI coding agent support
  • CI workflow updated
  • npm audit fix applied to dependencies (high vulnerability issues were being reported)

Testing

  • 55+ new test files under src/tests/api/ organized by resource, one spec per action (e.g., projects/list.spec.ts, runs/create.spec.ts)
  • Each spec typically contains: mocked tests with MSW handlers, validation error tests, and live tests tagged with { tags: ['live'] }
  • Shared test infrastructure in test-helper.ts with useMockServer(), runCli(), and a test fixture that provides project lifecycle management
  • Global setup (global-setup.ts) handles live API authentication
  • Missing subcommand help tests ensure incomplete commands show help text

🤖 Generated with Claude Code

@gemini-code-assist
Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a new api command to the QAS CLI, providing direct access to the QA Sphere public API. This enhancement allows users to manage their QA Sphere projects, test runs, and results programmatically from the command line. The PR also includes significant documentation updates and comprehensive testing to ensure the new functionality is robust and easy to use.

Highlights

  • API Command: Introduces a new api command for direct access to the QA Sphere public API, enabling users to manage projects, test runs, and results programmatically.
  • Documentation: Extensive documentation updates in README.md and a new SKILL.md file to support AI coding agents, providing detailed command usage and examples.
  • Testing: Adds comprehensive tests for the new api command, including both mocked and live modes, ensuring reliability and correctness.
Ignored Files
  • Ignored by pattern: .github/workflows/** (1)
    • .github/workflows/ci.yml
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

gemini-code-assist[bot]

This comment was marked as outdated.

claude[bot]

This comment was marked as outdated.

claude[bot]

This comment was marked as outdated.

claude[bot]

This comment was marked as outdated.

claude[bot]

This comment was marked as outdated.

claude[bot]

This comment was marked as outdated.

claude[bot]

This comment was marked as outdated.

claude[bot]

This comment was marked as outdated.

claude[bot]

This comment was marked as outdated.

claude[bot]

This comment was marked as outdated.

- Add Zod schemas at API level for all request types with validateRequest()
  that throws RequestValidationError, caught by apiHandler for clean error output
- Add handleValidationError() + buildArgumentMap() to map API field names to
  CLI argument names (e.g., title → --title) for user-friendly error messages
- Move validation schemas from command-level to API-level as single source of
  truth; command handlers pass args directly to API functions
- Add customFields, parameterValues, and filledTCaseTitleSuffixParams to
  test-case create/update commands with proper validation (parameterValues
  restricted to template type)
- Fix jsonResponse null crash, expand ResultStatus with custom statuses,
  relax test-cases update check, extract shared superRefine for run schemas
- Add tests for new fields (mocked + live) and batch-create validation errors
claude[bot]

This comment was marked as outdated.

@ramilamparo ramilamparo marked this pull request as ready for review March 25, 2026 09:06
claude[bot]

This comment was marked as outdated.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e484fb2a50

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

src/api/run.ts Outdated
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed the file to runs.ts to keep it uniform with the others.

skills/SKILL.md Outdated
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New skill can be added with npx skills add Hypersequent/qas-cli

Comment on lines +60 to +67
- name: Run live API tests
env:
QAS_TEST_URL: ${{ secrets.QAS_TEST_URL }}
QAS_TEST_TOKEN: ${{ secrets.QAS_TEST_TOKEN }}
QAS_TEST_USERNAME: ${{ secrets.QAS_TEST_USERNAME }}
QAS_TEST_PASSWORD: ${{ secrets.QAS_TEST_PASSWORD }}
QAS_DEV_AUTH: ${{ secrets.QAS_DEV_AUTH }}
run: npm run test:live
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Live tests are added that allows running automated tests with a real live qasphere instance. QAS_TEST_USERNAME and QAS_TEST_PASSWORD are required because we don't have a public API yet for deleting projects, which is required for clean-ups after the tests. So we use login credentials and call a non-public API instead.

"typescript": "^5.8.3",
"typescript-eslint": "^8.31.1",
"vitest": "^3.1.2"
"vitest": "^4.1.0"
Copy link
Copy Markdown
Collaborator Author

@ramilamparo ramilamparo Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated vitest to allow specifying tags. Which is useful for live tests and excluding them on normal testing.

Copy link
Copy Markdown
Collaborator

@satvik007 satvik007 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was testing with claude and found some bugs

⏺ Bugs Found

  1. Trailing slash on QAS_URL breaks all API commands — QAS_URL="https://host.com/" produces double-slash URLs
    (https://host.com//api/...) returning 404 HTML. createApi() doesn't strip trailing slashes like
    ResultUploadCommandHandler does.
  2. folders bulk-create --folders expects wrong JSON format — Help says '[{"path": ["Parent", "Child"]}]'
    (array), but validation requires '{"folders": [{"path": ["Parent", "Child"]}]}' (object wrapper).
  3. test-cases list --include help suggests invalid values — Help example says "tags,requirements" but server
    rejects these with a validation error. Valid values are undocumented in the CLI.
  4. runs tcases list --limit silently ignored — Passing --limit 1 still returns all 20 test cases. The CLI
    accepts the arg but the API endpoint doesn't paginate.
  5. test-cases list --page 0 crashes with raw Go error — Server requires page >= 1 but CLI doesn't validate.
    Error shown: Field validation for 'Page' failed on the 'gt' tag.
  6. settings update-statuses color help/example are wrong — Help says "color": "#FF9800" (hex), but API
    requires named colors: blue, gray, red, orange, yellow, green, teal, indigo, purple, pink.

skills/SKILL.md Outdated
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reorganize this skill to be more brief.
AI should be able to ask for help on particular command to know more details.
Otherwise, at the very least repeating the headers of the table for each API doesn't seem like a good idea.
Maybe I am wrong about this, and it doesn't matter because of context lengths of new models.

Copy link
Copy Markdown
Collaborator Author

@ramilamparo ramilamparo Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the command tables with a tree instead. According to claude the skill takes up 1800-2000 tokens.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its possible to also move the api files inside the folders to have code that changes frequently lives together.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what you mean?

README.md Outdated
Comment on lines +102 to +114
## Command: `api`

The `api` command provides direct access to the QA Sphere public API from the command line. Outputting JSON to stdout for easy scripting and piping.

```
qasphere api <resource> <action> [options]
```

### Conventions

- **JSON output** — All commands print JSON to stdout; errors go to stderr.
- **`--project-code`** — Specified per-command (not global). Some endpoints (audit-logs, users, settings) are org-scoped and don't need it.
- **Comma-separated arrays** — Options like `--milestone-ids`, `--tags`, `--priorities`, `--folders` accept comma-separated values: `--tags 1,2,3`.
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this documentation is too dense. We can have detailed documentation on docs.qasphere.com but here we should just write the commands without much explanation because. a lof of it should be self-explanatory.

e.g. qasphere api custom-fields list is going to list custom fields.
Explaining this is not necessary here.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced the markdown tables with a tree instead.

Comment on lines +3 to +10
export default {
// Reusable fields shared across commands
'project-code': `Project code identifying the QA Sphere project.`,

list: {
command: 'List all tags in a project.',
epilog: apiDocsEpilog('tag', 'list-project-tags'),
},
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't the command example also show the project argument?

% ./build/bin/qasphere.js api tags --help
qasphere.js api tags

Manage tags

Commands:
  qasphere.js api tags list  List all tags in a project.

Options:
  -h, --help     Show help  [boolean]
      --version  Show version number  [boolean]
      --verbose  Show verbose errors  [boolean]
apple@Apples-MacBook-Pro qas-cli % ./build/bin/qasphere.js api tags list
Missing required argument: project-code

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

qasphere api tags will just show the available sub commands. The subcommand list is needed to show full argument options qasphere api tags list -h.

claude[bot]

This comment was marked as outdated.

@ramilamparo
Copy link
Copy Markdown
Collaborator Author

ramilamparo commented Mar 27, 2026

@satvik007

I was testing with claude and found some bugs

⏺ Bugs Found

1. Trailing slash on QAS_URL breaks all API commands — QAS_URL="https://host.com/" produces double-slash URLs
   (https://host.com//api/...) returning 404 HTML. createApi() doesn't strip trailing slashes like
   ResultUploadCommandHandler does.

Fixed.

2. folders bulk-create --folders expects wrong JSON format — Help says '[{"path": ["Parent", "Child"]}]'
   (array), but validation requires '{"folders": [{"path": ["Parent", "Child"]}]}' (object wrapper).

Fixed. Argument should now only accept an array.

3. test-cases list --include help suggests invalid values — Help example says "tags,requirements" but server
   rejects these with a validation error. Valid values are undocumented in the CLI.

Updated help text to include possible values.

4. runs tcases list --limit silently ignored — Passing --limit 1 still returns all 20 test cases. The CLI
   accepts the arg but the API endpoint doesn't paginate.

Looks like the API doesn't pagination --limit. I've removed the limit argument.

5. test-cases list --page 0 crashes with raw Go error — Server requires page >= 1 but CLI doesn't validate.
   Error shown: Field validation for 'Page' failed on the 'gt' tag.

Could not reproduce. Possibly you've tested older code?

6. settings update-statuses color help/example are wrong — Help says "color": "#FF9800" (hex), but API
   requires named colors: blue, gray, red, orange, yellow, green, teal, indigo, purple, pink.

Fixed

@claude
Copy link
Copy Markdown

claude bot commented Mar 27, 2026

Claude finished @ramilamparo's task in 9m 13s —— View job


Reviewing PR #58: Add public API commands

  • Gather PR context and diff
  • Launch specialized review subagents (code quality, security, performance, test coverage)
  • Aggregate findings, deduplicate, and post review
  • Clean up outdated bot comments

Reviewed the full diff (~420K chars, 100 files) with 4 specialized subagents (code quality, security, performance, test coverage). All 12 previously raised issues remain resolved. The latest 3 commits address reviewer feedback well (specific validators, step schema fix, --include fix, precondition support).

1 minor issue posted as inline comment (files upload missing file existence check for user-friendly errors).

1 still-open thread from prior review: precondition-text/precondition-id missing from updateFields in test-cases create/update handlers.

Cleanup: Minimized 13 outdated bot review summaries, deleted 4 progress comments.

claude[bot]

This comment was marked as outdated.

@claude
Copy link
Copy Markdown

claude bot commented Mar 27, 2026

Claude finished @ramilamparo's task in 11m 14s —— View job


Reviewing PR #58: Add public API commands

  • Gather PR context and diff
  • Read review instructions
  • Launch specialized review subagents (code quality, security, performance, test coverage)
  • Fetch existing comments for deduplication
  • Aggregate findings, deduplicate, and post review
  • Clean up outdated bot comments

Reviewed the full diff (~424K chars, 100 files) with 4 specialized subagents (code quality, security, performance, test coverage). All 12 previously raised issues from prior reviews remain resolved.

1 new finding posted as inline comment: withHttpRetry honors unbounded Retry-After values with no cap, allowing a misconfigured server to hang the CLI indefinitely.

2 still-open threads from prior reviews:

  1. precondition-text/precondition-id missing from updateFields in test-cases create/update handlers
  2. files upload missing file existence check for user-friendly errors

Cleanup: Minimized 14 outdated bot review summaries.

claude[bot]

This comment was marked as outdated.

@claude
Copy link
Copy Markdown

claude bot commented Mar 27, 2026

Claude finished @ramilamparo's task in 4m 21s —— View job


Reviewing PR #58: Add public API commands

  • Gather PR context and diff
  • Launch specialized review subagents (code quality, security, performance, test coverage)
  • Fetch existing comments for deduplication
  • Aggregate findings, deduplicate, and post review
  • Clean up outdated bot comments and progress comments

Reviewed the full diff (~424K chars, 100 files) with 4 specialized subagents. All 12 previously raised issues from prior reviews remain resolved. Posted 2 new inline comments and minimized 15 outdated bot review summaries.

New findings:

  1. withHttpRetry honors unbounded Retry-After values — can hang CLI indefinitely
  2. ArgumentValidationError.name set to wrong value (nit)

2 still-open threads from prior reviews:

  • precondition-text/precondition-id missing from updateFields in test-cases handlers
  • files upload missing file existence check for user-friendly errors

claude[bot]

This comment was marked as outdated.

@claude
Copy link
Copy Markdown

claude bot commented Mar 27, 2026

Reviewing PR #58: Add public API commands

  • Gather PR context and diff
  • Launch specialized review subagents (code quality, security, performance, test coverage)
  • Fetch existing comments for deduplication
  • Aggregate findings, deduplicate, and post review
  • Clean up outdated bot comments and progress comments

View job run

Copy link
Copy Markdown

@claude claude bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary

Reviewed the full diff (~424K chars, 100 files) with 4 specialized subagents (code quality, security, performance, test coverage). All 12 previously raised issues from prior reviews have been resolved. No new critical, high, or medium issues found.

2 still-open minor threads from prior reviews:

  1. precondition-text/precondition-id missing from updateFields / buildArgumentMap in both test-cases create (line 311) and update (line 433) handlers — affects error message completeness and validation error path mapping
  2. files upload missing file existence check — readFileSync gives raw Node.js ENOENT instead of a user-friendly error

Both are low-severity quality improvements, not blockers.

What's been fixed well across the 28 commits:

  • Path parameter injection mitigated via validateResourceId/validateProjectCode/validateIntId
  • ResultStatus type now includes custom statuses
  • DRY: body-merging extracted to mergeBodyWithOverrides, query plan validation to shared validateQueryPlans
  • listRunTCases response unwrapping fixed
  • test-cases update check rewritten for flexibility
  • Trailing slash on QAS_URL handled
  • .int().positive() added to numeric array schemas
  • jsonResponse null-safe check added

@claude
Copy link
Copy Markdown

claude bot commented Mar 27, 2026

Claude finished @ramilamparo's task in 3m 41s —— View job


Reviewing PR #58: Add public API commands

  • Gather PR context and diff
  • Launch specialized review subagents (code quality, security, performance, test coverage)
  • Fetch existing comments for deduplication
  • Aggregate findings, deduplicate, and post review
  • Clean up outdated bot comments and progress comments

Reviewed the full diff (~424K chars, 100 files) with 4 specialized subagents (code quality, security, performance, test coverage). All 12 previously raised issues from prior reviews have been resolved. Posted 2 new inline comments and minimized 16 outdated bot review summaries.

New findings:

  1. shared-preconditions.ts:26 — Return type doesn't handle null that the API returns for empty lists (confirmed by live test)
  2. runs.ts:11import type for type-only Folder import (nit)

2 still-open threads from prior reviews:

  • precondition-text/precondition-id missing from updateFields in test-cases handlers
  • files upload missing file existence check for user-friendly errors

Copy link
Copy Markdown

@claude claude bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary

Reviewed the full diff (~424K chars, 100 files) with 4 specialized subagents (code quality, security, performance, test coverage). All 12 previously raised issues from prior reviews have been resolved.

2 new inline comments posted:

  1. shared-preconditions.ts:26 — Return type doesn't handle null that the API returns for empty lists (confirmed by live test)
  2. runs.ts:11import type for type-only Folder import (nit)

2 still-open threads from prior reviews:

  1. precondition-text/precondition-id missing from updateFields in test-cases create/update handlers — affects error messages and validation error mapping
  2. files upload missing file existence check — readFileSync gives raw Node.js ENOENT instead of user-friendly error

Test coverage observation (non-blocking): Validation error paths remain lightly tested for several commands (settings update-statuses, folders bulk-create, test-plans create, files upload, results create). The runs create and test-cases create specs are good models for coverage.

No security or performance issues found.

const validated = params ? validateRequest(params, ListSharedPreconditionsRequestSchema) : {}
return fetcher(
appendSearchParams(`/api/public/v0/project/${projectCode}/shared-precondition`, validated)
).then((r) => jsonResponse<SharedPrecondition[]>(r))
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Return type doesn't account for null — live test confirms API returns null for empty lists

The live test at src/tests/api/shared-preconditions/list.spec.ts:49 types the result as SharedPrecondition[] | null and comments "API returns null for empty list". But the API function here types the return as SharedPrecondition[] without handling null.

If the CLI calls printJson(result) on a null return, it prints null instead of []. More importantly, any TypeScript caller trusting the return type will assume it's always an array and could crash on .length, .map(), etc.

Suggested fix:

.then((r) => jsonResponse<SharedPrecondition[] | null>(r))
.then((r) => r ?? [])

This is consistent with how other list endpoints handle nullable responses (e.g., projects.list() uses .then((r) => r.projects ?? [])).

sortOrderParam,
validateRequest,
} from './schemas'
import { Folder } from './folders'
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: import type for type-only import

Folder is used only as a type annotation (in the RunTCase interface). The project convention is to use import type for type-only imports (e.g., src/tests/api/shared-preconditions/list.spec.ts:3).

import type { Folder } from './folders'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants