Implement observed asymptotic CLs likelihood limit#136
Open
davidwalter2 wants to merge 1 commit into
Open
Conversation
The observed likelihood-based upper limit was never produced: the call in
rabbit_limit.py was commented out ("TODO: make it work"), the channel/mapping
case was a `pass`, and the old compute_likelihood_limit had a Jacobian bug
(it differentiated sqrt(2*NLL) instead of sqrt(2*(NLL-min)), mis-scaling the
constraint Jacobian fed to the joint trust-constr solve).
Rewrite compute_likelihood_limit to reuse the same profiled-NLL machinery as
the (working) expected limit: evaluate the profile-likelihood test statistic
with the data fit (q_mu) and the background-only Asimov fit (q_{mu,A}) and
root-find mu over CLs(mu) = cl_s, using the same algebra as the gaussian path
(Cowan et al. arXiv:1007.1727). Add the modified statistic q-tilde_mu for a
negative best-fit signal strength, consistent with the gaussian xobs<0 branch.
- fitter.py: add Fitter.profiled_nll_at_poi (freeze/minimize/reduced_nll,
restoring state), the literal profile-likelihood test-statistic definition.
- asymptotic_limits.py: replace the buggy bespoke two-fitter solver with the
CLs root-find; handle the channel/mapping case by reporting the mapped
observable at the limit point.
- rabbit_limit.py: wire the observed likelihood limit for both the parameter
and channel cases; enable the likelihood_asymptoticLimits_observed output.
- main.yml: assert the observed likelihood limit is finite and within the
expected band in the bsm CI jobs (it was silently NaN before).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The observed likelihood-based asymptotic CLs upper limit was never produced:
rabbit_limit.pywas commented out (# TODO: make it work) and the channel/mapping case was apass(# TODO: implement for channels); thelikelihood_asymptoticLimits_observedoutput histogram was commented out — so the bsm CI jobs silently yielded NaN.compute_likelihood_limithad a real Jacobian bug: it differentiatedsqrt(2*NLL)instead ofsqrt(2*(NLL - nllvalreduced)), mis-scaling the constraint Jacobian fed to the jointtrust-constrsolve.Changes
Rewrite
compute_likelihood_limitto reuse the same profiled-NLL machinery as the (already working) expected limit, instead of the bespoke two-fitter joint solve: evaluate the profile-likelihood test statistic with the data fit (q_μ) and the background-only Asimov fit (q_{μ,A}) and root-find μ overCLs(μ) = cl_s, using the same algebra as the gaussian path (Cowan, Cranmer, Gross, Vitells, arXiv:1007.1727).rabbit/fitter.py: addFitter.profiled_nll_at_poi— fix the POI, profile all other parameters (freeze_params/minimize/reduced_nll), restore state. The literal profile-likelihood test-statistic definition.rabbit/asymptotic_limits.py: replace the buggy bespoke solver with the CLs root-find; implement the modified statistic q̃_μ (Eq. 16) for μ̂ < 0, consistent with the gaussianxobs < 0branch; handle the channel/mapping case by reporting the mapped observable at the limit point (same convention as the expected channel limit).bin/rabbit_limit.py: wire the observed likelihood limit for both the parameter and channel cases; enable thelikelihood_asymptoticLimits_observedoutput..github/workflows/main.yml: assert in both bsm CI jobs that the observed likelihood limit exists, is finite/positive, and sits within the expected ±2σ band (previously silently NaN).Verification
End-to-end on the existing bsm CI flow (
tests/make_tensor.py --bsm):bsm1bsm1_sig_mixingEach method's observed limit sits just below its own expected median (consistent with the slightly negative μ̂; modified statistic engaged).
isort/black/flake8/pylintclean, workflow YAML valid.🤖 Generated with Claude Code