Add num_trials trace to BenchmarkResult#4941
Open
saitcakmak wants to merge 1 commit intofacebook:mainfrom
Open
Add num_trials trace to BenchmarkResult#4941saitcakmak wants to merge 1 commit intofacebook:mainfrom
saitcakmak wants to merge 1 commit intofacebook:mainfrom
Conversation
|
@saitcakmak has exported this pull request. If you are a Meta employee, you can view the originating Diff in D93751894. |
saitcakmak
added a commit
to saitcakmak/Ax
that referenced
this pull request
Feb 24, 2026
Summary: The optimization_trace and other traces on BenchmarkResult are indexed by "completion event" rather than trial number. In the asynchronous case, multiple trials can complete at the same simulated time and get grouped into a single trace entry, so the trace length can be less than the number of trials. Previously there was no way to determine how many trials had completed at each trace entry. This diff adds a `num_trials` field to `BenchmarkResult` (a `list[int]`) that records the cumulative number of completed or early-stopped trials at each completion event. In the synchronous case this is simply `[1, 2, ..., n]`. In the async case it can increase by more than 1 at a step, e.g. `[2, 4, 5, ...]`. The field is optional (`None`) for backwards compatibility with old stored results. On `AggregatedBenchmarkResult`, the mean `num_trials` across replications is added as a new column on the `optimization_trace` and `score_trace` DataFrames (when available on all results). Reviewed By: hvarfner Differential Revision: D93751894
6d47967 to
f4cb6c1
Compare
saitcakmak
added a commit
to saitcakmak/Ax
that referenced
this pull request
Feb 24, 2026
Summary: The optimization_trace and other traces on BenchmarkResult are indexed by "completion event" rather than trial number. In the asynchronous case, multiple trials can complete at the same simulated time and get grouped into a single trace entry, so the trace length can be less than the number of trials. Previously there was no way to determine how many trials had completed at each trace entry. This diff adds a `num_trials` field to `BenchmarkResult` (a `list[int]`) that records the cumulative number of completed or early-stopped trials at each completion event. In the synchronous case this is simply `[1, 2, ..., n]`. In the async case it can increase by more than 1 at a step, e.g. `[2, 4, 5, ...]`. The field is optional (`None`) for backwards compatibility with old stored results. On `AggregatedBenchmarkResult`, the mean `num_trials` across replications is added as a new column on the `optimization_trace` and `score_trace` DataFrames (when available on all results). Reviewed By: hvarfner Differential Revision: D93751894
f4cb6c1 to
41b1cdf
Compare
saitcakmak
added a commit
to saitcakmak/Ax
that referenced
this pull request
Feb 24, 2026
Summary: Pull Request resolved: facebook#4941 The optimization_trace and other traces on BenchmarkResult are indexed by "completion event" rather than trial number. In the asynchronous case, multiple trials can complete at the same simulated time and get grouped into a single trace entry, so the trace length can be less than the number of trials. Previously there was no way to determine how many trials had completed at each trace entry. This diff adds a `num_trials` field to `BenchmarkResult` (a `list[int]`) that records the cumulative number of completed or early-stopped trials at each completion event. In the synchronous case this is simply `[1, 2, ..., n]`. In the async case it can increase by more than 1 at a step, e.g. `[2, 4, 5, ...]`. The field is optional (`None`) for backwards compatibility with old stored results. On `AggregatedBenchmarkResult`, the mean `num_trials` across replications is added as a new column on the `optimization_trace` and `score_trace` DataFrames (when available on all results). Reviewed By: hvarfner Differential Revision: D93751894
41b1cdf to
5914a65
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4941 +/- ##
=======================================
Coverage 96.80% 96.80%
=======================================
Files 595 595
Lines 63193 63213 +20
=======================================
+ Hits 61173 61194 +21
+ Misses 2020 2019 -1 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
saitcakmak
added a commit
to saitcakmak/Ax
that referenced
this pull request
Feb 24, 2026
Summary: The optimization_trace and other traces on BenchmarkResult are indexed by "completion event" rather than trial number. In the asynchronous case, multiple trials can complete at the same simulated time and get grouped into a single trace entry, so the trace length can be less than the number of trials. Previously there was no way to determine how many trials had completed at each trace entry. This diff adds a `num_trials` field to `BenchmarkResult` (a `list[int]`) that records the cumulative number of completed or early-stopped trials at each completion event. In the synchronous case this is simply `[1, 2, ..., n]`. In the async case it can increase by more than 1 at a step, e.g. `[2, 4, 5, ...]`. The field is optional (`None`) for backwards compatibility with old stored results. On `AggregatedBenchmarkResult`, the mean `num_trials` across replications is added as a new column on the `optimization_trace` and `score_trace` DataFrames (when available on all results). Reviewed By: hvarfner Differential Revision: D93751894
5914a65 to
a7f5d20
Compare
saitcakmak
added a commit
to saitcakmak/Ax
that referenced
this pull request
Feb 24, 2026
Summary: Pull Request resolved: facebook#4941 The optimization_trace and other traces on BenchmarkResult are indexed by "completion event" rather than trial number. In the asynchronous case, multiple trials can complete at the same simulated time and get grouped into a single trace entry, so the trace length can be less than the number of trials. Previously there was no way to determine how many trials had completed at each trace entry. This diff adds a `num_trials` field to `BenchmarkResult` (a `list[int]`) that records the cumulative number of completed or early-stopped trials at each completion event. In the synchronous case this is simply `[1, 2, ..., n]`. In the async case it can increase by more than 1 at a step, e.g. `[2, 4, 5, ...]`. The field is optional (`None`) for backwards compatibility with old stored results. On `AggregatedBenchmarkResult`, the mean `num_trials` across replications is added as a new column on the `optimization_trace` and `score_trace` DataFrames (when available on all results). Reviewed By: hvarfner Differential Revision: D93751894
a7f5d20 to
ff000f4
Compare
Summary: The optimization_trace and other traces on BenchmarkResult are indexed by "completion event" rather than trial number. In the asynchronous case, multiple trials can complete at the same simulated time and get grouped into a single trace entry, so the trace length can be less than the number of trials. Previously there was no way to determine how many trials had completed at each trace entry. This diff adds a `num_trials` field to `BenchmarkResult` (a `list[int]`) that records the cumulative number of completed or early-stopped trials at each completion event. In the synchronous case this is simply `[1, 2, ..., n]`. In the async case it can increase by more than 1 at a step, e.g. `[2, 4, 5, ...]`. The field is optional (`None`) for backwards compatibility with old stored results. On `AggregatedBenchmarkResult`, the mean `num_trials` across replications is added as a new column on the `optimization_trace` and `score_trace` DataFrames (when available on all results). Reviewed By: hvarfner Differential Revision: D93751894
ff000f4 to
975f98f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
The optimization_trace and other traces on BenchmarkResult are indexed by "completion event" rather than trial number. In the asynchronous case, multiple trials can complete at the same simulated time and get grouped into a single trace entry, so the trace length can be less than the number of trials. Previously there was no way to determine how many trials had completed at each trace entry.
This diff adds a
num_trialsfield toBenchmarkResult(alist[int]) that records the cumulative number of completed or early-stopped trials at each completion event. In the synchronous case this is simply[1, 2, ..., n]. In the async case it can increase by more than 1 at a step, e.g.[2, 4, 5, ...]. The field is optional (None) for backwards compatibility with old stored results.On
AggregatedBenchmarkResult, the meannum_trialsacross replications is added as a new column on theoptimization_traceandscore_traceDataFrames (when available on all results).Reviewed By: hvarfner
Differential Revision: D93751894