Open
Conversation
…tch_utils Summary: Renames the `max_parallelism` parameter to `max_concurrency` across GenerationStep, GenerationNode, and the generation strategy dispatch utilities. Adds backward-compatible deprecated `max_parallelism` parameters with deprecation warnings where the public API is affected (`choose_generation_strategy`). Internal variable names (`sobol_parallelism`, `bo_parallelism`) are renamed to `sobol_concurrency`, `bo_concurrency` for consistency. Differential Revision: D92457714
Summary: Renames the `parallelism` parameter to `concurrency` in `Client.run_trials()` and adds backward-compatible deprecated `max_parallelism` parameters in `AxClient.create_experiment()` and `AxClient.get_max_parallelism()` → `get_max_concurrency()`. Both include deprecation warnings guiding callers to use the new parameter names, with validation that old and new parameters are not specified simultaneously. Differential Revision: D93771849
…Settings Summary: Renames `num_parallel_jobs` to `num_concurrent_jobs` in `BenchmarkExecutionSettings` and all nightly benchmark configurations. Also updates the docstring in `BenchmarkMethod` to reference "pending trials" instead of "parallelism". This is a mechanical rename with no behavioral change. Differential Revision: D93771883
…ants, and telemetry Summary: Updates remaining references from "parallelism" to "concurrency" across orchestration, telemetry, early stopping, and other modules. This covers docstrings, comments, constant names (`MAX_PENDING_TRIALS` → `MAX_CONCURRENT_TRIALS`, `DUMMY_MAX_PENDING_TRIALS` → `DUMMY_MAX_CONCURRENT_TRIALS`), telemetry field names, and variable names in test files. No behavioral changes — purely a terminology alignment. Differential Revision: D93771906
…tDesign.concurrency_limit` Summary: As titled, adding a simple `ExperimentDesign` object. Putting it into properties for serialization for now, so as to not do duplicate work ahead of the storage refactor implementation (and also in case we change things while working on this stack). Differential Revision: D89770462
Summary: Migrates all references from `experiment._properties[Keys.EXPERIMENT_TOTAL_CONCURRENT_ARMS]` to `experiment.design.concurrency_limit`, completing the transition to the `ExperimentDesign` dataclass introduced in the prior diff. This affects generation node input constructors (including `ALL_N` and `REPEAT_N`), the Axolotl updater, and associated tests. Also cleans up the `no-commit` code in `generation_node_input_constructors.py` to use the new `concurrency_limit` field with a fallback to a default of 10. Differential Revision: D89772029
Summary:
## Changes
Consolidates `generate_candidates` and `_prepare_trials` into a unified API:
- Renames `generate_candidates` → `generate_candidate_trials` and changes its return type to a 3-tuple `(existing_candidates, new_trials, error)`, incorporating the existing-candidate-trial logic that was previously in `_prepare_trials`.
- Extracts the capacity/limit calculation from `_prepare_trials` into a new `compute_n_to_generate` method, which the Orchestrator's main loop now calls before `generate_candidate_trials`.
- Renames `should_generate_candidates_for_pts` → `should_generate_candidate_trials_for_pts` and adds a "not enough data" check that validates metrics have at least 1 day of data before allowing generation.
- Adds two new test methods for the "not enough data" and "missing metrics + not enough data" scenarios.
## Devmate session
How doing this with Devmate went:
1. First we ask Devmate to analyse the difference betwen the methods; it does remarkably well:{F1984363089} {F1984363089} {F1984363089}
2. Next a tangent: I renamed `generate_candidates` with a more precise name (`generate_candidate_trials`), since that is the method we will keep between the two, and it might as well have a better name. Asked Devmate to apply the changes throughout fbcode.
{F1984363157} {F1984363170}
3. Now for the hard part: get `generate_candidate_trials` to match the behavior or `_prepare_trials`, without me writing any of the code: {F1984363323} {F1984363333}
^ Pretty good for starters! I give corrections, see above; it applies them well: {F1984363346}
Then with one more small correction, we have a very solid plan: {F1984363398}, which Devmate implements: {F1984363406} {F1984363458}. I think it did really well!
Differential Revision: D89750211
Summary: Adds an `on_generation_error` callback parameter to `generate_candidate_trials` and a corresponding `on_generation_error` function in Axolotl utils. This allows callers like Axolotl to format error messages (including paste upload with full traceback) without the Orchestrator needing to know about paste infrastructure. The `generate_candidate_trials` return type changes from a 3-tuple to a 4-tuple, adding a `cannot_generate_reason` string that the callback populates when generation fails. The existing candidate trial accounting is also moved from `generate_candidate_trials` into `compute_n_to_generate`, so the `n` parameter now represents exactly the number of new trials to generate. Differential Revision: D89751541
Summary: Moves generation exception handling and trial creation from the Orchestrator into the GenerationStrategy, consolidating the flow and reducing the Orchestrator's complexity. Key changes: - Adds `GenerationStrategy.gen_handle_exceptions`, which wraps `gen()` and catches known non-fatal exceptions (`OptimizationComplete`, `DataRequiredError`, `MaxParallelismReachedException`, `AxGenerationException`, `OptimizationConfigRequired`), returning them as structured reason strings instead of raising. - Inlines the trial creation logic (previously in `_get_next_trials` and `_gen_new_trials_from_generation_strategy`) directly into `generate_candidate_trials`, removing both helper methods entirely (~130 lines removed). - Removes a now-unnecessary test in `test_ax_sweep_orchestrator` that tested the deleted `_gen_new_trials_from_generation_strategy` method. Differential Revision: D89754292
meta-cla
bot
added
the
CLA Signed
|
@mgrange1998 has exported this pull request. If you are a Meta employee, you can view the originating Diff in D89754292. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Moves generation exception handling and trial creation from the Orchestrator into the GenerationStrategy, consolidating the flow and reducing the Orchestrator's complexity.
Key changes:
GenerationStrategy.gen_handle_exceptions, which wrapsgen()and catches known non-fatal exceptions (OptimizationComplete,DataRequiredError,MaxParallelismReachedException,AxGenerationException,OptimizationConfigRequired), returning them as structured reason strings instead of raising._get_next_trialsand_gen_new_trials_from_generation_strategy) directly intogenerate_candidate_trials, removing both helper methods entirely (~130 lines removed).test_ax_sweep_orchestratorthat tested the deleted_gen_new_trials_from_generation_strategymethod.Differential Revision: D89754292