write function that translates from hubverse format to scoringutils-ready format

[nickreich]

Create a user-accessible function to handle transforming model-output data to a scoringutil forecast object: https://epiforecasts.io/scoringutils/dev/reference/as_forecast.html

Essentially, the goal of this function is to ensure that model output data is converted to the corresponding scoringutil format by calling the appropriate internal transformation function (these functions are defined in #22, #23, #24, #25, #26).

For example, model output data of output_type median would be passed to the internal hubEvals transform_point_model_output function.

This function should be exported for a user who wants more direct, custom access to scoringutils functionality, but also will be called in hubEvals::score_mdl_output() (#13)

migrated from epiforecasts/scoringutils#605

Definition of done:

• Create function
• Unit test(s)
• Update documentation (vignettes, news, etc.)

[nickreich]

To clarify what distinguishes this issue from #12, I think the idea is that this function would be exported and accessible by users where the functions specified in #12 would not be.

[elray1]

From Sam, see also:

in epinowcast for example it looks like this: https://github.com/epinowcast/epinowcast/blob/7a9ebcd972d8098bce7fe1067f221c19721873af/R/model-validation.R#L139

used here (scroll up): https://package.epinowcast.org/dev/articles/germany-age-stratified-nowcasting.html#summary

the vision here is something like the broom package.

Look here for inspiration about how to set this up.

[seabbs]

I'm happy to have a first pass at this if you would like to assign me? Probably next week.

[elray1]

That would be great! I'll assign you now.

Just to add some updates to what's in the issue statement above, the package already has three non-exported functions that are related to this: transform_pmf_model_out, transform_point_model_out, and transform_quantile_model_out. So one statement of the goal here is to figure out the right way to create public access to those functions. Maybe this involves renaming those or creating new S3 methods for objects of S3 class model_out_tbl?

[seabbs]

Maybe this involves renaming those or creating new S3 methods for objects of S3 class model_out_tbl?

My expectation would be this

[seabbs]

A slight wrinkle is the differences in approach to defining output type.

i.e. here

hubEvals/R/validate.R

Line 78 in 040d000

validate_output_type <- function(model_out_tbl) {

as I think the data format here is a standard data frame this isn't going to be play nicely with our s3 dispatch unless a special hubeval or other hub format class is available. There are obvious pluses and minuses to having that in the ecosystem but it seems like maybe the question of a hub format class is not for here?

[elray1]

We have the model_out_tbl class that's defined in hubUtils.

The current setup for hubEvals is that we accept a standard data frame and cast it to model_out_tbl if necessary during validations. Right now, those validations are called from within the individual transform_*_model_out functions, e.g. here. And in turn, those transform_*_model_out functions are called from score_model_out.

So I think in principle this could work with a setup using S3 methods because we have an appropriate class to use, but it will require a bit of reshuffling to do that conversion to model_out_tbl before we call the transformation functions. Not sure, but this may require splitting up the content of validate_model_oracle_out into parts that are really related to just checking that we can convert the inputs to model_out_tbl and parts that are related to checking compatibility of the model_out_tbl and the oracle_output.

As an aside, the model_out_tbl class allows for representations of multiple output types (quantile, median, etc.) in one data frame. So the validation function you linked to above was checking that, for purposes of linking to scoringutils, the provided model_out_tbl in fact only contains a single output type.

[seabbs]

Nice makes sense.

As an aside, the model_out_tbl class allows for representations of multiple output types (quantile, median, etc.) in one data frame.

Yes this is slightly different to how we have been thinking about things (as we have subclasses) so might take a bit of fiddling as well. I'll take a look at the code properly and propose some kind of plan here next week.