Derive MRRA via gradient descent

[EshitaJoshi]

Use per-mirror d80 measurements to derivemirror_reflection_random_angle via gradient descent.

1. Gradients are computed via central finite differences for each RNDA parameter.

   • $\sigma_1$ and $\sigma_2$ are scaled in log space because the values are very small $\sim 10^{-3}$ deg
   • frac is scaled linearly as it ranges from 0 to 1

2. Gradients are clipped to avoid instability due to very large updates.

   • Update steps are bounded (MAX_LOG_STEP, MAX_FRAC_STEP) and all parameters are clipped to their schema-defined bounds.

3. Adaptive learning rate is used

   • If the objective improves, the learning rate is increased slightly.
   • If not, the step is rejected and the learning rate is reduced.

4. Optimization stops when either:

   • The absolute percentage difference falls below the given threshold, or
   • The learning rate becomes negligible (1e-12), or
   • The maximum number of iterations is reached (100)

Usually takes around ~10 iterations per mirror to converge.

To run use:

simtools-derive-mirror-rnda --test --site North --telescope LSTN-01 --model_version 7.0.0 --data tests/resources/MLTdata-preproduction.ecsv --parameter_version 1.0.0 --d80_hist --cleanup

The per-mirror measured and simulated d80 distributions look like:

The final averaged value of the parameter is exported like this in the telescope sub-folder: mirror_reflection_random_angle-1.0.0.json

and the per-mirror optimization is exported like this:
per_mirror_rnda.json

The algorithm is parallelised, using the --n_workers flag or os.cpu_count() by default.

The parallelisation is generalised and moved to simtools.job_execution. Closes #1990.

[Copilot]

Pull Request Overview

This PR implements gradient descent optimization for deriving the mirror_reflection_random_angle parameter in Monte Carlo simulations for gamma-ray astronomy. The implementation replaces the previous iterative comparison-based approach with a more robust RMSD-based gradient descent optimization.

• Extends the existing PSF parameter optimization framework to support optimizing only mirror_reflection_random_angle
• Introduces gradient descent optimization with adaptive learning rates and configurable RMSD thresholds (default 3%)
• Adds final PSF comparison visualization showing optimized parameters and fit quality
• Replaces D80 containment-based optimization with full PSF curve RMSD minimization

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
src/simtools/ray_tracing/psf_parameter_optimisation.py	Extended PSFParameterOptimizer to support selective parameter optimization, added epsilon parameter to gradient descent methods, and added support for single mirror mode in ray tracing
src/simtools/ray_tracing/mirror_panel_psf.py	Refactored to use gradient descent optimization instead of iterative D80 comparison, removed old optimization logic, updated result printing to show RMSD instead of D80
src/simtools/applications/derive_mirror_rnda.py	Updated command-line interface to support gradient descent parameters (threshold, learning_rate), removed old PSF measurement arguments, added cleanup option for intermediate files
src/simtools/visualization/plot_psf.py	Enhanced plot functions to handle optional parameter groups, added create_final_psf_comparison_plot for final optimization results
tests/unit_tests/ray_tracing/test_psf_parameter_optimisation.py	Added comprehensive tests for gradient descent workflow, refactored tests to use helper fixtures, added tests for single mirror and full telescope modes
tests/unit_tests/ray_tracing/test_mirror_panel_psf.py	Replaced old optimization tests with gradient descent tests, added tests for various mirror modes and random focal length configurations
tests/unit_tests/visualization/test_plot_psf.py	Added test for create_final_psf_comparison_plot function
docs/changes/1911.feature.md	Added changelog entry for gradient descent implementation

[ctao-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[GernotMaier]

Minor comment - please implement. I approve already. Thanks!!

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 6 comments.

[GernotMaier]

Very nice! I can now understand what is happening. A couple of minor comments, nothing big so I approve.

For the plotting: is there still functionality to plot? The important point is to have all plotting related in modules of visualize, but we still want that functionality. Did I miss it?

[GernotMaier]

Is ok - but I feel unwell about removing files with "*"

[ctao-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[EshitaJoshi]

Thanks for the many detailed reviews @GernotMaier! I have made the changes you listed, the main change was just renaming everything from d80 to psf and using --fraction as a configurable parameter. There is still plotting functionality which has been moved to the visualize/plot_psf.py module. I will merge this PR now, thanks for the approval!

[GernotMaier]

Please merge - thanks for the patience!