as_forecast_sample.<package> adapters home

[seabbs]

This is a discussion issue for identifying where we think package adapters to the scoringutils format should live.

The design intention here is that developers with or without our support would implement adapters for their formats into the scoringutils format and then signpost to our docs for how to work with it. This is nice as it makes there be a standard scoringutils API vs an ecosystem with potentially lots of adapters with different names, or, worse, copy and pasted code from vignettes, and a range of vignettes demonstrating scoringutils with a range of practice that may get out of date (or even worse a wrapper around score itself!)

The issue with this design is that to implement a converter package, maintainers have to depend on scoringutils (this is a hard requirement of extending a s3 method). For some packages, this isn't a huge issue as scoringutils is quite lightweight but for others I think it could be a real problem. Aside from that, there is also the perception of having an additional dependency being a problem, which we might like to avoid?

So what do we do about this?

One option is that we add converters from other package formats to scoringutils itself. Here as we are the home of the s3 generic this could be done as a suggests and so not impact our dependency tree. The downside though, is it adds code for us to support (i.e. if the package format changes the converter may break).

Another option is we spin out the s3 generic parts of the package into a lightweight package that we then depend on. This could then be a zero upstream dependency for others. However, in may cases they may want to use our tools i.e as_forecast_sample.dataframe etc which would lead back to the core package as a dependency.

A final option is we suggest maintainers make their own extension packages for scoringutils adapters if they don't want us as a dependency

I think the several packages have hit this issue i.e broom, parsnip, tidymodels etc. so understanding what they do to resolve it seems like it might be a useful path.

There are several examples of these converters in the wild:

• https://github.com/epinowcast/epinowcast/blob/ac58a21937d07e7d5a2edbe93ea3ba72aae63f5f/R/model-validation.R#L32 (which has a supporting vignette to show usage)
• Add format parameter to get_predictions() for scoringutils-compatible output EpiNow2#1262 here @sbfnk implements an internal format that is compatible with as_forecast_sample vs having a converter. This is only practical due to developer overlap (i.e. agreement on the format). If we ever changed the scoringutils format EpiNow2 would need to make internal breaking changes.
• https://github.com/hubverse-org/hubEvals/blob/89b3c64d7db7e47fc1e7f3c3186bf2f73aed21bb/R/score_model_out.R#L192 hubevals implements a almost complete reimplementation of scoringutils in the name of them having a consistent UI.
• Consider adding an evaluation example into one of the vignettes epinowcast/baselinenowcast#418 the initial desire here was to implement a manual converter in a vignette with a full scoring workflow.

[nikosbosse]

Thanks for opening this! We should definitely document this. Do we have concrete examples in the wild of people being unwilling to depend on scoringutils? If so, I would try to get in contact with them and look for solutions. Creating new lightweight packages feels like adding extra complexity. I don't particularly mind taking on the burden of adding connectors for selected packages, but I agree with the general sentiment (it not being ideal)

[seabbs]

I am unwilling to depend on it in epinowcast, baselinenowcast etc. for what is ultimately downstream functionality. @sbfnk avoided this by adding a manual internal conversion that works with the default as_forecast_sample method.

[seabbs]

I don't think others are sufficiently aware that they could do this or that they might want to have thought about it hugely but could be wrong.

[nikosbosse]

Ah I see. Well that is clear user demand then :)
And I guess suggests doesn't work for S3? Could scoringutils shed any dependencies that would make you willing again to depend on it? I guess data.table is a relevant dependency for people who aren't using it in their packages...
Unsure, I don't think I have strong preferences. I'll ask Claude to do some research on best practices

[nikosbosse]

High-level summary:
apparently there are 2 broad options, according to Claude

• Option A: Lightweight generics package (the broom/generics pattern)
• Option B: Delayed S3 method registration (the vctrs pattern)

Claude thinks delayed registration should just work. Does that work?

---

CLAUDE: Research on how other R package ecosystems solve the S3 generic extension / dependency problem:

How the tidyverse solved this: the generics package

broom originally defined tidy(), glance(), augment() directly, meaning any package wanting a tidy() method had to Import broom and all its dependencies. To fix this, r-lib created the generics package — an extremely lightweight package containing only generic function definitions.

• Dependency footprint: Essentially zero (only methods, which is base R)
• How it works: broom re-exports tidy() from generics. Downstream packages (like recipes, broom.mixed) Import generics instead of broom, getting access to the S3 generics without the heavy dependency tree.
• Contents: ~30+ generics including tidy, glance, augment, fit, generate, etc.
• Proven pattern — on CRAN since 2018, used across the entire tidymodels ecosystem.

Alternative: Delayed S3 method registration (no new package needed)

Since R 3.6.0, R natively supports delayed S3 method registration. A downstream package can define a method for a generic from a Suggested (not Imported) package:

Using roxygen2 @exportS3Method:

#' @exportS3Method scoringutils::as_forecast_sample
as_forecast_sample.epinowcast <- function(x, ...) {
  # conversion implementation
}

This generates S3method(scoringutils::as_forecast_sample, epinowcast) in NAMESPACE. The method is only registered when scoringutils is actually loaded. scoringutils goes in Suggests, not Imports.

Using vctrs::s3_register() in .onLoad():

Alternatively, copy the standalone s3_register() function from rlang (explicitly designed to be vendored) via:

usethis::use_standalone("r-lib/rlang", "s3-register")

Then in .onLoad():

.onLoad <- function(libname, pkgname) {
  s3_register("scoringutils::as_forecast_sample", "epinowcast")
}

This is how vctrs registers methods for pillar and tibble generics without depending on them.

Two options for scoringutils

Option A: Lightweight generics package (the broom/generics pattern)

Create a tiny package (e.g. forecastgenerics) containing only as_forecast_sample(), as_forecast_quantile(), etc. as S3 generics with zero dependencies. scoringutils re-exports from it; downstream packages Import only the lightweight package.

Pros: Clean explicit dependency, normal method dispatch, compile-time checking.
Cons: Another package to maintain, coordination overhead for adding new generics, would need to define shared classes/constructors too.

Option B: Downstream packages use delayed registration (the vctrs pattern)

Downstream packages list scoringutils in Suggests and use @exportS3Method or s3_register().

Pros: No new package needed, zero changes to scoringutils, well-supported since R 3.6, each downstream decides independently.
Cons: Methods only available when scoringutils is installed+loaded, downstream authors need to know about this pattern.

Recommendation

For the current ecosystem size (4-5 downstream packages), Option B (delayed registration) seems like the right starting point — it requires zero changes to scoringutils and is now the standard R mechanism. The concrete recipe for downstream packages is:

1. Add scoringutils to Suggests in DESCRIPTION
2. Define methods with @exportS3Method scoringutils::as_forecast_sample
3. Done — the method registers automatically when a user loads scoringutils

If the ecosystem grows significantly, extracting generics into a dedicated lightweight package (Option A) can be done later as a non-breaking change.

The key insight is that this is a solved problem in R — downstream packages don't need to Import scoringutils to provide as_forecast_sample methods. They just need to use delayed registration, which is a one-line roxygen tag.

Sources

• generics package / GitHub
• rlang standalone-s3-register.R
• roxygen2 namespace docs
• R Blog: S3 Method Lookup (2019)
• Epiverse-TRACE: S3 generics post

[seabbs]

The generic approach I think won't work as you end up needing our tooling here when defining a new method.

It sounds like there maybe have been some steps forward here in base and maybe in rlang which I wasn't aware of. If those work then we can just provide some guidance and it is all good in the hood!

[seabbs]

I'm looking into this now

[seabbs-bot]

We've just opened a PR in epinowcast to move scoringutils from Imports to Suggests: epinowcast/epinowcast#741

The approach uses @exportS3Method scoringutils::as_forecast_sample (roxygen2 >= 7.2.0) which generates a S3method(scoringutils::as_forecast_sample, epinowcast) directive in NAMESPACE. R handles the delayed S3 method registration natively — if scoringutils is loaded, the method dispatches as expected; if not, the package still loads and passes R CMD check cleanly.

This was straightforward to implement and could serve as a reference for other packages wanting to provide scoringutils adapters without taking a hard dependency.

This was opened by a bot. Please ping @seabbs for any questions.

[nikosbosse]

Sweet! So in the end we just need to document this, right?

[seabbs]

It looks like it!