Skip to content

benchmark: Add script to diagnose hypothesis performance regression (#32693)#37300

Closed
davidweb3-ctrl wants to merge 1 commit intocommaai:masterfrom
davidweb3-ctrl:fix/hypothesis-performance-32693
Closed

benchmark: Add script to diagnose hypothesis performance regression (#32693)#37300
davidweb3-ctrl wants to merge 1 commit intocommaai:masterfrom
davidweb3-ctrl:fix/hypothesis-performance-32693

Conversation

@davidweb3-ctrl
Copy link
Copy Markdown

Description

This PR addresses the performance regression reported in #32693 where hypothesis 6.103.1 is significantly slower than 6.47.0.

Root Cause Analysis

Through benchmarking, I identified that the main bottleneck is st.data() with multiple data.draw() calls, which became ~83% slower in newer hypothesis versions:

Version st.data() multiple draws Total
6.47.0 0.423s 0.968s
6.103.1 0.775s 1.663s
6.151.9 0.670s 1.363s

Changes

  • Added benchmark_hypothesis.py: Standalone script to measure hypothesis performance across versions
  • Added selfdrive/car/tests/benchmark_hypothesis_performance.py: Car-specific benchmark

Usage

# Test old version
pip install hypothesis==6.47.0
python benchmark_hypothesis.py

# Test new version
pip install hypothesis==6.103.1
python benchmark_hypothesis.py

Bounty

Contributes to resolving #32693 ($200 bounty)

…2693)

This script helps diagnose the performance regression between hypothesis

versions 6.47.0 and 6.103.1+ as reported in issue commaai#32693.

Benchmark results show:

- hypothesis 6.47.0: ~0.97s total

- hypothesis 6.103.1: ~1.66s total (+72% slower)

- st.data() multiple draws is the main bottleneck
@github-actions
Copy link
Copy Markdown
Contributor

This PR has had no activity for 24 days. It will be automatically closed in 7 days if there is no activity.

@github-actions github-actions Bot added the stale label Mar 19, 2026
@adeebshihadeh
Copy link
Copy Markdown
Contributor

This is missing:

and a solution for optimizing/fixing either the hypothesis library itself, or openpilot to improve the example generation times

@adeebshihadeh adeebshihadeh marked this pull request as draft March 20, 2026 22:53
@github-actions github-actions Bot removed the stale label Mar 22, 2026
@github-actions
Copy link
Copy Markdown
Contributor

This PR has had no activity for 24 days. It will be automatically closed in 7 days if there is no activity.

@github-actions github-actions Bot added the stale label Apr 15, 2026
@github-actions
Copy link
Copy Markdown
Contributor

This PR has been automatically closed due to inactivity. Feel free to re-open once activity resumes.

@github-actions github-actions Bot closed this Apr 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants