Merge branch 'develop' into test-metrics-point

epiforecasts · Jan 5, 2024 · 10641d6 · 10641d6
2 parents 8c35bec + e3bf253
commit 10641d6
Show file tree

Hide file tree

Showing 42 changed files with 205 additions and 76 deletions.
diff --git a/.Rbuildignore b/.Rbuildignore
@@ -18,3 +18,4 @@
 ^\.devcontainer$
 ^CODE_OF_CONDUCT\.md$
 ^inst/manuscript/output$
+^CRAN-SUBMISSION$
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
@@ -0,0 +1,21 @@
+<!-- Thanks for opening this pull request! Below we have provided a suggested template for PRs to this repository and a checklist to complete before opening a PR -->
+
+## Description
+
+This PR closes #<issue-number>.
+
+[Describe the changes that you made in this pull request.]
+
+## Checklist
+
+- [ ] My PR is based on a package issue and I have explicitly linked it.
+- [ ] I have included the target issue or issues in the PR title as follows: *issue-number*: PR title
+- [ ] I have tested my changes locally.
+- [ ] I have added or updated unit tests where necessary.
+- [ ] I have updated the documentation if required.
+- [ ] I have built the package locally and run rebuilt docs using roxygen2.
+- [ ] My code follows the established coding standards and I have run `lintr::lint_package()` to check for style issues introduced by my changes. 
+- [ ] I have added a news item linked to this PR.
+- [ ] I have reviewed CI checks for this PR and addressed them as far as I am able.
+
+<!-- Thanks again for this PR - @scoringutils dev team -->
diff --git a/.gitignore b/.gitignore
@@ -18,3 +18,4 @@ docs
 ..bfg-report/
 .DS_Store
 .vscode
+README.html
diff --git a/DESCRIPTION b/DESCRIPTION
@@ -41,8 +41,8 @@ Description:
     Scoring metrics can be used either through a convenient data.frame format, 
     or can be applied as individual functions in a vector / matrix format. 
     All functionality has been implemented with a focus on performance and is 
-    robustly tested. Find more information about scoringutils in the 
-    accompanying paper (Bosse et al., 2022) <arXiv:2205.07090v1>. 
+    robustly tested. Find more information about the package in the 
+    accompanying paper (<doi:10.48550/arXiv.2205.07090>).
 License: MIT + file LICENSE
 Encoding: UTF-8
 LazyData: true

diff --git a/NEWS.md b/NEWS.md
@@ -39,15 +39,26 @@ The update introduces breaking changes. If you want to keep using the older vers
 - `score()` now calls `na.omit()` on the data, instead of only removing rows with missing values in the columns `observed` and `predicted`. This is because `NA` values in other columns can also mess up e.g. grouping of forecasts according to the unit of a single forecast.
 - added documentation for the return value of `summarise_scores()`.
 - Removed abs_error and squared_error from the package in favour of `Metrics::ae` and `Metrics::se`. 
+- Added unit tests for `interval_coverage_quantile()` and `interval_coverage_dev_quantile()` in order to make sure that the functions provide the correct warnings when insufficient quantiles are provided.
+
+# scoringutils 1.2.2
+
+## Package updates
+- `scoringutils` now depends on R 3.6. The change was made since packages `testthat` and `lifecycle`, which are used in `scoringutils` now require R 3.6. We also updated the Github action CI check to work with R 3.6 now. 
+- Added a new PR template with a checklist of things to be included in PRs to facilitate the development and review process
+
+## Bug fixes
+- Fixes a bug with `set_forecast_unit()` where the function only workded with a data.table, but not a data.frame as an input. 
+- The metrics table in the vignette [Details on the metrics implemented in `scoringutils`](https://epiforecasts.io/scoringutils/articles/metric-details.html) had duplicated entries. This was fixed by removing the duplicated rows. 
 
 # scoringutils 1.2.1
 
 ## Package updates
 - This minor update fixes a few issues related to gh actions and the vignettes displayed at epiforecasts.io/scoringutils. It
-  - gets rid of the preferably package in _pkgdown.yml. The theme had a toggle between light and dark theme that didn't work properly
-  - updates the gh pages deploy action to v4 and also cleans up files when triggered
-  - introduces a gh action to automatically render the Readme from Readme.Rmd
-  - removes links to vignettes that have been renamed
+  - Gets rid of the preferably package in _pkgdown.yml. The theme had a toggle between light and dark theme that didn't work properly
+  - Updates the gh pages deploy action to v4 and also cleans up files when triggered
+  - Introduces a gh action to automatically render the Readme from Readme.Rmd
+  - Removes links to vignettes that have been renamed
 
 # scoringutils 1.2.0
 

diff --git a/R/available_forecasts.R b/R/available_forecasts.R
@@ -29,7 +29,9 @@
 #' @export
 #' @keywords check-forecasts
 #' @examples
-#' data.table::setDTthreads(1) # only needed to avoid issues on CRAN
+#' \dontshow{
+#'   data.table::setDTthreads(2) # restricts number of cores used on CRAN
+#' }
 #'
 #' get_forecast_counts(example_quantile,
 #'   by = c("model", "target_type")

diff --git a/R/data.R b/R/data.R
@@ -20,7 +20,7 @@
 #'   \item{horizon}{forecast horizon in weeks}
 #' }
 # nolint start
-#' @source \url{https://github.com/covid19-forecast-hub-europe/covid19-forecast-hub-europe/commit/a42867b1ea152c57e25b04f9faa26cfd4bfd8fa6/}
+#' @source \url{https://github.com/european-modelling-hubs/covid19-forecast-hub-europe/commit/a42867b1ea152c57e25b04f9faa26cfd4bfd8fa6/}
 # nolint end
 "example_quantile"
 
@@ -47,7 +47,7 @@
 #'   \item{horizon}{forecast horizon in weeks}
 #' }
 # nolint start
-#' @source \url{https://github.com/covid19-forecast-hub-europe/covid19-forecast-hub-europe/commit/a42867b1ea152c57e25b04f9faa26cfd4bfd8fa6/}
+#' @source \url{https://github.com/european-modelling-hubs/covid19-forecast-hub-europe/commit/a42867b1ea152c57e25b04f9faa26cfd4bfd8fa6/}
 # nolint end
 "example_point"
 
@@ -74,7 +74,7 @@
 #'   \item{sample_id}{id for the corresponding sample}
 #' }
 # nolint start
-#' @source \url{https://github.com/covid19-forecast-hub-europe/covid19-forecast-hub-europe/commit/a42867b1ea152c57e25b04f9faa26cfd4bfd8fa6/}
+#' @source \url{https://github.com/european-modelling-hubs/covid19-forecast-hub-europe/commit/a42867b1ea152c57e25b04f9faa26cfd4bfd8fa6/}
 # nolint end
 "example_continuous"
 
@@ -101,7 +101,7 @@
 #'   \item{sample_id}{id for the corresponding sample}
 #' }
 # nolint start
-#' @source \url{https://github.com/covid19-forecast-hub-europe/covid19-forecast-hub-europe/commit/a42867b1ea152c57e25b04f9faa26cfd4bfd8fa6/}
+#' @source \url{https://github.com/european-modelling-hubs/covid19-forecast-hub-europe/commit/a42867b1ea152c57e25b04f9faa26cfd4bfd8fa6/}
 # nolint end
 "example_integer"
 
@@ -134,7 +134,7 @@
 #'   \item{predicted}{predicted value}
 #' }
 # nolint start
-#' @source \url{https://github.com/covid19-forecast-hub-europe/covid19-forecast-hub-europe/commit/a42867b1ea152c57e25b04f9faa26cfd4bfd8fa6/}
+#' @source \url{https://github.com/european-modelling-hubs/covid19-forecast-hub-europe/commit/a42867b1ea152c57e25b04f9faa26cfd4bfd8fa6/}
 # nolint end
 "example_binary"
 
@@ -159,7 +159,7 @@
 #'   \item{horizon}{forecast horizon in weeks}
 #' }
 # nolint start
-#' @source \url{https://github.com/covid19-forecast-hub-europe/covid19-forecast-hub-europe/commit/a42867b1ea152c57e25b04f9faa26cfd4bfd8fa6/}
+#' @source \url{https://github.com/european-modelling-hubs/covid19-forecast-hub-europe/commit/a42867b1ea152c57e25b04f9faa26cfd4bfd8fa6/}
 # nolint end
 "example_quantile_forecasts_only"
 
@@ -181,7 +181,7 @@
 #'   \item{location_name}{name of the country for which a prediction was made}
 #' }
 # nolint start
-#' @source \url{https://github.com/covid19-forecast-hub-europe/covid19-forecast-hub-europe/commit/a42867b1ea152c57e25b04f9faa26cfd4bfd8fa6/}
+#' @source \url{https://github.com/european-modelling-hubs/covid19-forecast-hub-europe/commit/a42867b1ea152c57e25b04f9faa26cfd4bfd8fa6/}
 # nolint end
 "example_truth_only"
 

diff --git a/R/metrics-quantile.R b/R/metrics-quantile.R
@@ -241,7 +241,8 @@ interval_coverage_quantile <- function(observed, predicted, quantile, range = 50
   if (!all(necessary_quantiles %in% quantile)) {
     warning(
       "To compute the interval coverage for a range of ", range,
-      "%, the quantiles ", necessary_quantiles, " are required. Returning `NA`."
+      "%, the quantiles `", toString(necessary_quantiles),
+      "` are required. Returning `NA`."
     )
     return(NA)
   }

diff --git a/R/pairwise-comparisons.R b/R/pairwise-comparisons.R
@@ -51,7 +51,9 @@
 #' @author Johannes Bracher, \email{johannes.bracher@@kit.edu}
 #' @keywords scoring
 #' @examples
-#' data.table::setDTthreads(1) # only needed to avoid issues on CRAN
+#' \dontshow{
+#'   data.table::setDTthreads(2) # restricts number of cores used on CRAN
+#' }
 #'
 #' scores <- score(example_quantile)
 #' pairwise <- pairwise_comparison(scores, by = "target_type")

diff --git a/R/pit.R b/R/pit.R
@@ -62,7 +62,9 @@
 #' @seealso [pit()]
 #' @importFrom stats runif
 #' @examples
-#' data.table::setDTthreads(1) # only needed to avoid issues on CRAN
+#' \dontshow{
+#'   data.table::setDTthreads(2) # restricts number of cores used on CRAN
+#' }
 #'
 #' ## continuous predictions
 #' observed <- rnorm(20, mean = 1:20)

diff --git a/R/plot.R b/R/plot.R
@@ -24,7 +24,9 @@
 #' @examples
 #' library(ggplot2)
 #' library(magrittr) # pipe operator
-#' data.table::setDTthreads(1) # only needed to avoid issues on CRAN
+#' \dontshow{
+#'   data.table::setDTthreads(2) # restricts number of cores used on CRAN
+#' }
 #'
 #' scores <- score(example_quantile) %>%
 #'   summarise_scores(by = c("model", "target_type")) %>%
@@ -577,11 +579,12 @@ make_na <- make_NA
 #' @importFrom data.table dcast
 #' @export
 #' @examples
-#' data.table::setDTthreads(1) # only needed to avoid issues on CRAN
+#' \dontshow{
+#'   data.table::setDTthreads(2) # restricts number of cores used on CRAN
+#' }
 #' data_coverage <- add_coverage(example_quantile)
 #' summarised <- summarise_scores(data_coverage, by = c("model", "range"))
 #' plot_interval_coverage(summarised)
-
 plot_interval_coverage <- function(scores,
                                    colour = "model") {
   ## overall model calibration - empirical interval coverage
@@ -830,7 +833,9 @@ plot_pairwise_comparison <- function(comparison_result,
 #' @importFrom stats density
 #' @return vector with the scoring values
 #' @examples
-#' data.table::setDTthreads(1) # only needed to avoid issues on CRAN
+#' \dontshow{
+#'   data.table::setDTthreads(2) # restricts number of cores used on CRAN
+#' }
 #'
 #' # PIT histogram in vector based format
 #' observed <- rnorm(30, mean = 1:30)

diff --git a/R/score.R b/R/score.R
@@ -32,7 +32,9 @@
 #' @importFrom stats na.omit
 #' @examples
 #' library(magrittr) # pipe operator
-#' data.table::setDTthreads(1) # only needed to avoid issues on CRAN
+#' \dontshow{
+#'   data.table::setDTthreads(2) # restricts number of cores used on CRAN
+#' }
 #'
 #' validated <- as_forecast(example_quantile)
 #' score(validated) %>%

diff --git a/R/summarise_scores.R b/R/summarise_scores.R
@@ -28,7 +28,9 @@
 #' to the names of the columns of the original data specified in `by` or
 #' `across` using the `fun` passed to `summarise_scores()`.
 #' @examples
-#' data.table::setDTthreads(1) # only needed to avoid issues on CRAN
+#' \dontshow{
+#'   data.table::setDTthreads(2) # restricts number of cores used on CRAN
+#' }
 #' library(magrittr) # pipe operator
 #' \dontrun{
 #' scores <- score(example_continuous)

diff --git a/R/zzz.R b/R/zzz.R
@@ -0,0 +1,9 @@
+.onAttach <- function(libname, pkgname) {
+  packageStartupMessage(
+    "Note: scoringutils is currently undergoing major development changes ",
+    "(with an update planned for the first quarter of 2024). We would very ",
+    "much appreciate your opinions and feedback on what should be included in ",
+    "this major update: ",
+    "https://github.com/epiforecasts/scoringutils/discussions/333"
+  )
+}
diff --git a/README.Rmd b/README.Rmd
@@ -30,7 +30,7 @@ library(knitr)
 
 The `scoringutils` package provides a collection of metrics and proper scoring rules and aims to make it simple to score probabilistic forecasts against observed values. 
 
-You can find additional information and examples in the papers [Evaluating Forecasts with scoringutils in R](https://arxiv.org/abs/2205.07090) [Scoring epidemiological forecasts on transformed scales](https://www.medrxiv.org/content/10.1101/2023.01.23.23284722v1) as well as the Vignettes ([Getting started](https://epiforecasts.io/scoringutils/articles/scoringutils.html), [Details on the metrics implemented](https://epiforecasts.io/scoringutils/articles/metric-details.html) and [Scoring forecasts directly](https://epiforecasts.io/scoringutils/articles/scoring-forecasts-directly.html)).
+You can find additional information and examples in the papers [Evaluating Forecasts with scoringutils in R](https://arxiv.org/abs/2205.07090) [Scoring epidemiological forecasts on transformed scales](https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1011393) as well as the Vignettes ([Getting started](https://epiforecasts.io/scoringutils/articles/scoringutils.html), [Details on the metrics implemented](https://epiforecasts.io/scoringutils/articles/metric-details.html) and [Scoring forecasts directly](https://epiforecasts.io/scoringutils/articles/scoring-forecasts-directly.html)).
 
 The `scoringutils` package offers convenient automated forecast evaluation through the function `score()`. The function operates on data.frames (it uses `data.table` internally for speed and efficiency) and can easily be integrated in a workflow based on `dplyr` or `data.table`. It also provides experienced users with a set of reliable lower-level scoring metrics operating on vectors/matrices they can build upon in other applications. In addition it implements a wide range of flexible plots designed to cover many use cases.