scoringutils development plan

[nikosbosse]

This issue lays out a development plan for scoringutils. This development plan is currently work in progress.

In general I think it makes sense to try and polish the current changes and work towards a new release soon. But since releasing soon means that we (likely) won't have all changes to the interface finalised, we might want to avoid releasing too many breaking changes to CRAN. My current thinking on this is to

• create a gh release in the next few weeks
• finalise the new interface before a new CRAN release. This could either mean thinking through the interface very carefully and making sure we are happy with everything, or skipping a major development version on CRAN.

Avoiding having to skip a major release on CRAN would definitely be nice. I'll go through all our issues to sort and prioritise them a bit and see whether that's realistic.

scoringutils 2.0.0 GitHub release

Release planned for: Monday, December 18, 2023

Aim and scope

Aim: Bringing the current development version into a state such that it fulfils the same functionality that version 1.2.0 had. We are not aiming to finalise all changes to the interface. The focus for the release is also not developing new futures, but rather on polishing current changes such that the package is usable with a new interface.

Depending on how attainable the goal of finalising all interface changes is, users version 2.0.0 should expect breaking changes or changes to the interface in the future. I think there is a bit of a tension between "getting the next version out of the door" and "addressing everything to avoid future changes".

To do before release

• CI fixes
  • fix pkgdown (Fix pkgdown  #482)
  • fix linting issues Fix linting issues #485
• Development
  • Create a print method for forecast_binary, forecast_quantile etc. #498
  • Create function quantile_coverage_deviation() #453
  • Discussion: Decide on the attributes scoringutils objects should have #477
• Fixes
  • Fix that validate() currently produces a silent warning for the example data #456
  • Improve way that we warn about a change in the forecast_unit #372
  • Testing: Make sure pit() and plot_pit() are correct #359
• Interface changes
  • Implement updated validation and class construction workflow #494
  • Meta-issue: Changes to add_pairwise_comparison() #517
  • Rename available_forecasts() to get_forecast_counts() #403
  • Rename add_coverage() to add_rowwise_coverage() #442
  • Implement consistent naming and language for talking about scoring rules #476
• Code refactoring
• Cleanup
  • Rethink metrics table #415
  • Make default metrics be the output of a function, rather than a stored data set #474
  • Documentation: Update Readme.rmd before release #439
  • Make sure functions are consistently called check_, test_, assert_, get_. #357
  • Implement consistent handling of the by argument #366
  • Meta Issue: Clean-up of current major changes #346
  • Rethink remove_scoringutils_class() #407
  • Documentation: Update / deduplicate docs for all functions before release #425
  • Rename coverage --> interval_coverage #436
  • Testing: Test everything that's user-facing with a data.frame instead of a data.table #438
  • Update Vignettes and documentation that relates to add_coverage() #496
  • Add examples for plotting functions back in #448
  • Delete library(data.table) in setup.R in tests #450
  • Rename apply_metrics() to apply_rules() and document it properly #452
  • Store whether scores have been summarised #461
• Documentation
  • Update News file
  • Documentation: Create a vignette that details the different inputs and internal workings of scoringutils #353

Things that are about unclear whether they are in score or not

• Potentially breaking changes

  • Create a wrapper around log score to warn about its use for integer-valued forecasts #360
  • Remake summarise_scores() into method for aggregate() #435
  • Discussion: Define implementation for metrics which work for different forecast types #338
  • Discussion: Should scoring rules always return a vector? #455
  • Discussion: Clean up various wis() and interval_score() functions #410 - needs some updates
  • Add check to validate_forecast() that predictions must not decrease with quantile #412
  • Rework plotting functionality #355
  • Create a default plot method for validated forecasts #414
  • Discussion: Rethink ... in `score() #431
  • Change scores to data as an argument for coverage plotting functions #497
  • Delete old scoring rules that used an interval format #443 (depending on what we call the functions etc)

• Nice to have, but unsure whether they are in scope or not:

  • Create standardised input checks for all non-scoring functions #352
  • Get rid of available_metrics() #383
  • Code refactoring as discussed in Meta-issue: Code refactoring #459
  • Documentation: Add more documentation + print method for default metrics #365
  • Feature: Improve handling of messages / warnings in score #418
  • Add more tests for score() #502

scoringutils 2.1.0 GitHub release

Aim and scope

Do more code refactoring (if not done in 2.0.0).
Add more features.

scoringutils 3.0.0 CRAN release

• Finalise changes to interface

scoringutils 3.0.0 CRAN release

Once the interface is finalised and stable, releasing to CRAN.

[seabbs]

create a gh release in the next few weeks

I think we need to make sure that the next release to main is then stable going forward so no massive breaking changes that we don't carefully manage depreciations for. That means making sure that we are happy with the overall workflow of the package.

I also think that we need to keep these as dev releases so that there are pkgdown docs available that work with the CRAN release of the package (or it will become very hard to use).

Release planned for: Monday, December 18, 2023

I think this might be a bit optimist and instead we should push into the new year and let people test out the develop branch to make sure we don't need to do any major refactoring to main.

The focus for the release is also not developing new futures, but rather on polishing current changes such that the package is usable with a new interface.

I definitely agree on this aim but think that we do want to make sure the new interface is stable. I would be more comfortable if @Bisaloo etc. had had time to do a review on the develop branch entire before pushing to main.

[seabbs]

users version 2.0.0 should expect breaking changes or changes to the interface in the future. I think there is a bit of a tension between "getting the next version out of the door" and "addressing everything to avoid future changes".

I think we should aim to avoid breaking changes post 2.0.0 and so if we thinking these are going to happen either manage depreciations for them post 2.0.0 release or hold off on the 2.0.0 release

[seabbs]

For things that are tagged as for future releases can we identify if they would require breaking changes or not?

[nikosbosse]

This issue has outlived it's usefulness a bit. I'm currently sorting issues into different projects/milestones and having a concrete pre-release issue for the next release as pointed out by Sam seems like a good idea.