Skip to content

Improve random state management#81

Closed
tautomer wants to merge 11 commits intoMSDLLCpapers:mainfrom
tautomer:fix_rng
Closed

Improve random state management#81
tautomer wants to merge 11 commits intoMSDLLCpapers:mainfrom
tautomer:fix_rng

Conversation

@tautomer
Copy link
Copy Markdown
Collaborator

@tautomer tautomer commented Aug 14, 2025

This PR addresses random state management issues in #80 to improve reproducibility.

Implementation

  • RNGManager class: Centralized wrapper managing NumPy and PyTorch generators with synchronized seeding
  • Single seed per campaign: One seed initialization ensures consistent randomness throughout campaign lifecycle
  • Explicit RNG passing: Components (Campaign, Optimizer, ExpDesigner, Simulator) accept RNG instances explicitly instead of global state resets
  • State persistence: Campaign and Optimizer save/load complete RNG state (seed + generator states)
  • Ownership tracking: _owns_rng flag distinguishes owned vs shared RNG instances
  • Temporary seed overrides: Context manager allows manual_seed parameter without global pollution
  • Backward compatibility: obsidian.USE_OLD_RNG_CONTROL toggle for legacy behavior
  • Comprehensive testing: 11 tests covering RNG lifecycle, state persistence, and reproducibility

Other changes

Some other improvements were made while this new RNG control was implemented.

  1. Introduced direct optimizer options passthrough to model fitting routine in campaign.fit. You can now do

    campaign.fit({"max_attempts": 10, "multi_starts": True})

    to fit with model with 10 different restarts.

  2. Introduced direct optimizer options passthrough to acquisition function optimization in optimizer.suggest

    campaign.optimizer.suggest(
        ...,
        optim_options={"method": "Powell", "maxiter": 100}
    )

    This will tell scipy.optimize.minimize to use Powell and maximum iteration of 100.

Current status

The following tests have been performed, both on single objective (modified from Simple single objective.ipynb)

  1. The new code, with obsidian.USE_OLD_RNG_CONTROL=True produces the same results as the current main branch.
  2. With new RNG control, two runs with the same initial random states achieve identical results with categorical parameters.
  3. With new RNG control, two runs with the same initial random states achieve identical results without categorical parameters. (fixed in 1064d6f).

Main changes in RNG:
1. Introduced GlobalRNG class for centralized RNG control
2. Replaced all instances of torch random number generation now with a generator
3. Replaced all instances of numpy.random with a generator
4. Introduced a with_tmp_seed context manager for all hidden/implicit random number generation
5. Introduced a package level toggle USE_OLD_RNG_CONTROL for backward compatibility

Misc fixes/improvements
1. Introduced direct optimizer options passthrough to model fitting routine in campaign.fit
2. Introduced direct optimizer options passthrough to acquisition function optimization in optimizer.suggest
3. Vectorized the noise generation in simulator
4. Some code style cleanups and type hint fixes around RNG-related blocks
@tautomer tautomer added bug Something isn't working enhancement New feature or request labels Aug 14, 2025
@tautomer tautomer linked an issue Aug 14, 2025 that may be closed by this pull request
@tautomer tautomer marked this pull request as draft August 14, 2025 22:23
@tautomer tautomer requested review from kstone40 and xuyuting August 14, 2025 22:23
1. Fix a regression from merging
2. Fix a bug where optimize_acqf get an input of fixed_features_list
@tautomer tautomer requested a review from spencermcminn August 25, 2025 21:36
1. Allow externally defined noise functions (can be related to X)
2. Make the Optithon benchmark function compatible with the Simulator
1. Drop the global RNG manager enforcement.
2. New default behavior: each Campaign creates its own RNG manager. This
   also paves the way for future dash app refactoring.
3. For temporary random state override, the seed is generated at call
   time instead of decorator definition time by default. This change
   leads to better reproducibility.
4. For numerical reproducbility, optimizer by default uses a fixed
   random state when every time fit or suggest is called.
5. Fix optimizer save and load state.
@tautomer tautomer marked this pull request as ready for review March 18, 2026 19:41
@tautomer tautomer requested review from fima5 and removed request for xuyuting March 18, 2026 19:41
- Always use RNG manager's seed if not set in campaign and optimizer
- Override seeds in test files when loading from legacy state dict
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RNG leads to repeated random numbers or non-reproducible results

1 participant