Skip to content

ci: Improve benchmarking tests with CodSpeed#236

Draft
Copilot wants to merge 2 commits intomainfrom
copilot/improve-benchmarking-tests
Draft

ci: Improve benchmarking tests with CodSpeed#236
Copilot wants to merge 2 commits intomainfrom
copilot/improve-benchmarking-tests

Conversation

Copy link
Contributor

Copilot AI commented Mar 25, 2026

Summary

Replace pytest-benchmark with pytest-codspeed for benchmarking, extend test coverage to full process lifecycle, and parameterize across connector types.

Changes

  • pyproject.toml: Swap pytest-benchmarkpytest-codspeed>=4.3.0
  • tests/benchmark/test_benchmarking.py:
    • test_benchmark_process_run — benchmarks init + run
    • test_benchmark_process_lifecycle — benchmarks full init → run → destroy
    • Both parameterized over AsyncioConnector and ZMQConnector
    • Uses @pytest.mark.benchmark decorator (CodSpeed-compatible)
  • .github/workflows/benchmarks.yaml: Replace manual checkout-both-branches-and-compare workflow with CodSpeedHQ/action@v4. Adds push to main and workflow_dispatch triggers for baseline tracking and backtesting. Uses OIDC auth (id-token: write).
  • uv.lock: Updated to reflect dependency change

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • astral.sh
    • Triggering command: /usr/bin/curl curl -LsSf REDACTED (dns block)
  • codspeed.io
    • Triggering command: /home/REDACTED/work/_temp/ghcca-node/node/bin/node /home/REDACTED/work/_temp/ghcca-node/node/bin/node --enable-source-maps /home/REDACTED/work/_temp/copilot-developer-action-main/dist/index.js (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Original prompt

This section details on the original issue you should resolve

<issue_title>ci: Improve benchmarking tests</issue_title>
<issue_description>### Summary

Use codspeed instead of the current solution for benchmarking tests, to provide better feedback in PRs.

Extend the benchmarking tests so that full process lifecycle is covered, as well as the existing test which just covers process run.

Add additional benchmark tests for the different connector types via parameterisation.

Example

No response

Alternatives

No response</issue_description>

<agent_instructions>See https://codspeed.io/docs/benchmarks/python for information on updating the Python tests.
Also make sure to replace the existing step in the Github action.</agent_instructions>

Comments on the Issue (you are @copilot in this section)


⚡ Quickly spin up Copilot coding agent tasks from anywhere on your macOS or Windows machine with Raycast.

- Replace pytest-benchmark with pytest-codspeed in test dependencies
- Rewrite benchmark tests to use @pytest.mark.benchmark decorator
- Add full process lifecycle benchmark (init + run + destroy)
- Parameterize benchmarks across AsyncioConnector and ZMQConnector
- Replace GitHub Actions workflow with CodSpeed integration

Co-authored-by: toby-coleman <13170610+toby-coleman@users.noreply.github.com>
Agent-Logs-Url: https://github.com/plugboard-dev/plugboard/sessions/b8d70f1d-7f6f-412b-9452-e70611a349de
Copilot AI changed the title [WIP] Enhance benchmarking tests using codspeed ci: Improve benchmarking tests with CodSpeed Mar 25, 2026
Copilot AI requested a review from toby-coleman March 25, 2026 07:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: Improve benchmarking tests

2 participants