Skip to content

Implement observed asymptotic CLs likelihood limit#136

Open
davidwalter2 wants to merge 1 commit into
WMass:mainfrom
davidwalter2:260516_observedLikelihoodLimit
Open

Implement observed asymptotic CLs likelihood limit#136
davidwalter2 wants to merge 1 commit into
WMass:mainfrom
davidwalter2:260516_observedLikelihoodLimit

Conversation

@davidwalter2
Copy link
Copy Markdown
Collaborator

Problem

The observed likelihood-based asymptotic CLs upper limit was never produced:

  • The call in rabbit_limit.py was commented out (# TODO: make it work) and the channel/mapping case was a pass (# TODO: implement for channels); the likelihood_asymptoticLimits_observed output histogram was commented out — so the bsm CI jobs silently yielded NaN.
  • The old compute_likelihood_limit had a real Jacobian bug: it differentiated sqrt(2*NLL) instead of sqrt(2*(NLL - nllvalreduced)), mis-scaling the constraint Jacobian fed to the joint trust-constr solve.
  • The modified statistic q̃_μ for a negative best-fit signal strength was a TODO.

Changes

Rewrite compute_likelihood_limit to reuse the same profiled-NLL machinery as the (already working) expected limit, instead of the bespoke two-fitter joint solve: evaluate the profile-likelihood test statistic with the data fit (q_μ) and the background-only Asimov fit (q_{μ,A}) and root-find μ over CLs(μ) = cl_s, using the same algebra as the gaussian path (Cowan, Cranmer, Gross, Vitells, arXiv:1007.1727).

  • rabbit/fitter.py: add Fitter.profiled_nll_at_poi — fix the POI, profile all other parameters (freeze_params/minimize/reduced_nll), restore state. The literal profile-likelihood test-statistic definition.
  • rabbit/asymptotic_limits.py: replace the buggy bespoke solver with the CLs root-find; implement the modified statistic q̃_μ (Eq. 16) for μ̂ < 0, consistent with the gaussian xobs < 0 branch; handle the channel/mapping case by reporting the mapped observable at the limit point (same convention as the expected channel limit).
  • bin/rabbit_limit.py: wire the observed likelihood limit for both the parameter and channel cases; enable the likelihood_asymptoticLimits_observed output.
  • .github/workflows/main.yml: assert in both bsm CI jobs that the observed likelihood limit exists, is finite/positive, and sits within the expected ±2σ band (previously silently NaN).

Verification

End-to-end on the existing bsm CI flow (tests/make_tensor.py --bsm):

case gaussian obs likelihood obs likelihood expected (median / ±2σ)
parameter bsm1 1.89 2.06 2.30 / [1.60, 3.78]
channel bsm1_sig_mixing 0.043 0.034 0.046 / [0.028, 0.091]

Each method's observed limit sits just below its own expected median (consistent with the slightly negative μ̂; modified statistic engaged). isort/black/flake8/pylint clean, workflow YAML valid.

🤖 Generated with Claude Code

The observed likelihood-based upper limit was never produced: the call in
rabbit_limit.py was commented out ("TODO: make it work"), the channel/mapping
case was a `pass`, and the old compute_likelihood_limit had a Jacobian bug
(it differentiated sqrt(2*NLL) instead of sqrt(2*(NLL-min)), mis-scaling the
constraint Jacobian fed to the joint trust-constr solve).

Rewrite compute_likelihood_limit to reuse the same profiled-NLL machinery as
the (working) expected limit: evaluate the profile-likelihood test statistic
with the data fit (q_mu) and the background-only Asimov fit (q_{mu,A}) and
root-find mu over CLs(mu) = cl_s, using the same algebra as the gaussian path
(Cowan et al. arXiv:1007.1727). Add the modified statistic q-tilde_mu for a
negative best-fit signal strength, consistent with the gaussian xobs<0 branch.

- fitter.py: add Fitter.profiled_nll_at_poi (freeze/minimize/reduced_nll,
  restoring state), the literal profile-likelihood test-statistic definition.
- asymptotic_limits.py: replace the buggy bespoke two-fitter solver with the
  CLs root-find; handle the channel/mapping case by reporting the mapped
  observable at the limit point.
- rabbit_limit.py: wire the observed likelihood limit for both the parameter
  and channel cases; enable the likelihood_asymptoticLimits_observed output.
- main.yml: assert the observed likelihood limit is finite and within the
  expected band in the bsm CI jobs (it was silently NaN before).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant