Variogram scores

[nickreich]

inspired by the recent epiengage meeting, I've been thinking about ways to get some standard scores that measure things like how well a model predicts relative rankings across locations. (The Allocation score does this, but with some additional baggage and needed assumptions.) It seems that maybe the variogram score (if i'm understanding correctly) might measure this, to some extent? I'd like to advocate for the ability to compute a VS on point and probabilistic forecasts (maybe you could only do this with joint samples for probabilistic forecasts), with the option to do it on transformed data. It seems that scoringRules has this implemented already.

[nickreich]

For point forecasts, it could be computed I think somewhat straightforwardly, although it would need a "grouping"(e.g., across a set of locations). For probabilistic forecasts, it would need sample indices that would be the same across that grouping.

[nickreich]

Also noting that I had Claude write up some simple examples to explore some limited properties of the variogram score, in part as it relates to a score like the alloscore:

• full notebook
• source code

[seabbs]

Like the notebook approach to thinking about this. Stealing it as a repo format.

[seabbs]

This looks good from a skim. Just digesting. For others the paper scoringRules cites is: https://journals.ametsoc.org/view/journals/mwre/143/4/mwr-d-14-00269.1.xml

[nickreich]

When does a discussion turn into a new issue/feature request?

[seabbs]

Aha well i was just going to read this and then promote it - so today!

[seabbs]

@jbracher I remember something about a quantile version of the energy score which would come at some of the concerns here from another direction. Is that a real memory?

[seabbs]

I made an issue and have added some implementation steps.

[seabbs]

Flagging: https://community.epinowcast.org/t/transform-and-aggregate-scoring-rules/383

Allen et al. seem to be arguing for doing more multivariate scoring via construction of univariate scores i.e transform, score, agg and flag that energy, variogram etc can be represented in this way. Seems like some stuff to think about here for furthering the good cause of finding that EpiNow2 default mode is really the only model we need.

[seabbs]

(though noting I am in agent space working on 8 parallel PRs which leads to a strange mindset).

[seabbs]

This is in place now! There are some issues for cleaning up etc but in principle good to go. The next step is working out what they should be used for and when but perhaps that isn't for this issue (so closing)

[nickreich]

re: what they should be used for... I'm working on a longer-term plan for increasing evaluation diversity in hubs and hub dashboards.

I realize I'm chiming in about this late, but can't the variogram score also be applied on point forecasts? I think as implemented now, it can only be applied with sample forecasts, right?

[seabbs]

Aha no I added a new multivariate_point class to support point scoring using them so we can do that with what is in place.

[seabbs]

(I think I did so correctly)

[nickreich]

awesome! thanks for the clarfication