Skip to content

Comments

Add num_trials trace to BenchmarkResult#4941

Open
saitcakmak wants to merge 1 commit intofacebook:mainfrom
saitcakmak:export-D93751894
Open

Add num_trials trace to BenchmarkResult#4941
saitcakmak wants to merge 1 commit intofacebook:mainfrom
saitcakmak:export-D93751894

Conversation

@saitcakmak
Copy link
Contributor

Summary:
The optimization_trace and other traces on BenchmarkResult are indexed by "completion event" rather than trial number. In the asynchronous case, multiple trials can complete at the same simulated time and get grouped into a single trace entry, so the trace length can be less than the number of trials. Previously there was no way to determine how many trials had completed at each trace entry.

This diff adds a num_trials field to BenchmarkResult (a list[int]) that records the cumulative number of completed or early-stopped trials at each completion event. In the synchronous case this is simply [1, 2, ..., n]. In the async case it can increase by more than 1 at a step, e.g. [2, 4, 5, ...]. The field is optional (None) for backwards compatibility with old stored results.

On AggregatedBenchmarkResult, the mean num_trials across replications is added as a new column on the optimization_trace and score_trace DataFrames (when available on all results).

Reviewed By: hvarfner

Differential Revision: D93751894

@meta-cla meta-cla bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Feb 24, 2026
@meta-codesync
Copy link

meta-codesync bot commented Feb 24, 2026

@saitcakmak has exported this pull request. If you are a Meta employee, you can view the originating Diff in D93751894.

saitcakmak added a commit to saitcakmak/Ax that referenced this pull request Feb 24, 2026
Summary:

The optimization_trace and other traces on BenchmarkResult are indexed by "completion event" rather than trial number. In the asynchronous case, multiple trials can complete at the same simulated time and get grouped into a single trace entry, so the trace length can be less than the number of trials. Previously there was no way to determine how many trials had completed at each trace entry.

This diff adds a `num_trials` field to `BenchmarkResult` (a `list[int]`) that records the cumulative number of completed or early-stopped trials at each completion event. In the synchronous case this is simply `[1, 2, ..., n]`. In the async case it can increase by more than 1 at a step, e.g. `[2, 4, 5, ...]`. The field is optional (`None`) for backwards compatibility with old stored results.

On `AggregatedBenchmarkResult`, the mean `num_trials` across replications is added as a new column on the `optimization_trace` and `score_trace` DataFrames (when available on all results).

Reviewed By: hvarfner

Differential Revision: D93751894
saitcakmak added a commit to saitcakmak/Ax that referenced this pull request Feb 24, 2026
Summary:

The optimization_trace and other traces on BenchmarkResult are indexed by "completion event" rather than trial number. In the asynchronous case, multiple trials can complete at the same simulated time and get grouped into a single trace entry, so the trace length can be less than the number of trials. Previously there was no way to determine how many trials had completed at each trace entry.

This diff adds a `num_trials` field to `BenchmarkResult` (a `list[int]`) that records the cumulative number of completed or early-stopped trials at each completion event. In the synchronous case this is simply `[1, 2, ..., n]`. In the async case it can increase by more than 1 at a step, e.g. `[2, 4, 5, ...]`. The field is optional (`None`) for backwards compatibility with old stored results.

On `AggregatedBenchmarkResult`, the mean `num_trials` across replications is added as a new column on the `optimization_trace` and `score_trace` DataFrames (when available on all results).

Reviewed By: hvarfner

Differential Revision: D93751894
saitcakmak added a commit to saitcakmak/Ax that referenced this pull request Feb 24, 2026
Summary:
Pull Request resolved: facebook#4941

The optimization_trace and other traces on BenchmarkResult are indexed by "completion event" rather than trial number. In the asynchronous case, multiple trials can complete at the same simulated time and get grouped into a single trace entry, so the trace length can be less than the number of trials. Previously there was no way to determine how many trials had completed at each trace entry.

This diff adds a `num_trials` field to `BenchmarkResult` (a `list[int]`) that records the cumulative number of completed or early-stopped trials at each completion event. In the synchronous case this is simply `[1, 2, ..., n]`. In the async case it can increase by more than 1 at a step, e.g. `[2, 4, 5, ...]`. The field is optional (`None`) for backwards compatibility with old stored results.

On `AggregatedBenchmarkResult`, the mean `num_trials` across replications is added as a new column on the `optimization_trace` and `score_trace` DataFrames (when available on all results).

Reviewed By: hvarfner

Differential Revision: D93751894
@codecov-commenter
Copy link

codecov-commenter commented Feb 24, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.80%. Comparing base (052a554) to head (975f98f).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4941   +/-   ##
=======================================
  Coverage   96.80%   96.80%           
=======================================
  Files         595      595           
  Lines       63193    63213   +20     
=======================================
+ Hits        61173    61194   +21     
+ Misses       2020     2019    -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

saitcakmak added a commit to saitcakmak/Ax that referenced this pull request Feb 24, 2026
Summary:

The optimization_trace and other traces on BenchmarkResult are indexed by "completion event" rather than trial number. In the asynchronous case, multiple trials can complete at the same simulated time and get grouped into a single trace entry, so the trace length can be less than the number of trials. Previously there was no way to determine how many trials had completed at each trace entry.

This diff adds a `num_trials` field to `BenchmarkResult` (a `list[int]`) that records the cumulative number of completed or early-stopped trials at each completion event. In the synchronous case this is simply `[1, 2, ..., n]`. In the async case it can increase by more than 1 at a step, e.g. `[2, 4, 5, ...]`. The field is optional (`None`) for backwards compatibility with old stored results.

On `AggregatedBenchmarkResult`, the mean `num_trials` across replications is added as a new column on the `optimization_trace` and `score_trace` DataFrames (when available on all results).

Reviewed By: hvarfner

Differential Revision: D93751894
saitcakmak added a commit to saitcakmak/Ax that referenced this pull request Feb 24, 2026
Summary:
Pull Request resolved: facebook#4941

The optimization_trace and other traces on BenchmarkResult are indexed by "completion event" rather than trial number. In the asynchronous case, multiple trials can complete at the same simulated time and get grouped into a single trace entry, so the trace length can be less than the number of trials. Previously there was no way to determine how many trials had completed at each trace entry.

This diff adds a `num_trials` field to `BenchmarkResult` (a `list[int]`) that records the cumulative number of completed or early-stopped trials at each completion event. In the synchronous case this is simply `[1, 2, ..., n]`. In the async case it can increase by more than 1 at a step, e.g. `[2, 4, 5, ...]`. The field is optional (`None`) for backwards compatibility with old stored results.

On `AggregatedBenchmarkResult`, the mean `num_trials` across replications is added as a new column on the `optimization_trace` and `score_trace` DataFrames (when available on all results).

Reviewed By: hvarfner

Differential Revision: D93751894
Summary:

The optimization_trace and other traces on BenchmarkResult are indexed by "completion event" rather than trial number. In the asynchronous case, multiple trials can complete at the same simulated time and get grouped into a single trace entry, so the trace length can be less than the number of trials. Previously there was no way to determine how many trials had completed at each trace entry.

This diff adds a `num_trials` field to `BenchmarkResult` (a `list[int]`) that records the cumulative number of completed or early-stopped trials at each completion event. In the synchronous case this is simply `[1, 2, ..., n]`. In the async case it can increase by more than 1 at a step, e.g. `[2, 4, 5, ...]`. The field is optional (`None`) for backwards compatibility with old stored results.

On `AggregatedBenchmarkResult`, the mean `num_trials` across replications is added as a new column on the `optimization_trace` and `score_trace` DataFrames (when available on all results).

Reviewed By: hvarfner

Differential Revision: D93751894
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed Do not delete this pull request or issue due to inactivity. fb-exported meta-exported

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants