Merge pull request #452 from nci/minor_release_8_4

Minor release 0.8.4

.binder/requirements.txt

@@ -281,7 +281,7 @@ referencing==0.34.0
     #   jsonschema
     #   jsonschema-specifications
     #   jupyter-events
-requests==2.31.0
+requests==2.32.2
     # via jupyterlab-server
 rfc3339-validator==0.1.4
     # via

.github/pull_request_template.md

@@ -20,7 +20,7 @@ Please work through the following checklists. Delete anything that isn't relevan
 - [ ] Short introduction to why you would use that metric and what it tells you
 - [ ] A link to a reference
 - [ ] A "things to try next" section at the end
-- [ ] Add notebook to [Explanation.ipynb](https://github.com/nci/scores/blob/develop/tutorials/Explanation.ipynb)
+- [ ] Add notebook to [Tutorial_Gallery.ipynb](https://github.com/nci/scores/blob/develop/tutorials/Tutorial_Gallery.ipynb)
 - [ ] Optional - a detailed discussion of how the metric works at the end of the notebook
 
 ## Documentation

README.md

@@ -1,12 +1,16 @@
 # Scores: Verification and Evaluation for Forecasts and Models
+
+> **Notice:**
+> **`scores` is undergoing final testing and review. When this is completed, this notice will be removed.**
+
 > 
 > **A list of over 50 metrics, statistical techniques and data processing tools contained in `scores` is [available here](https://scores.readthedocs.io/en/latest/included.html).**
 
 `scores` is a Python package containing mathematical functions for the verification, evaluation, and optimisation of forecasts, predictions or models. It primarily supports the meteorological, climatological and geoscience communities. In addition to supporting the Earth system science communities, it also has wide potential application in machine learning and other domains.
 
 Documentation is hosted at [scores.readthedocs.io](https://scores.readthedocs.io).  
 Source code is hosted at [github.com/nci/scores](https://github.com/nci/scores).  
-The tutorial gallery is hosted at [as part of the documentation, here](https://scores.readthedocs.io/en/latest/tutorials/Explanation.html).
+The tutorial gallery is hosted at [as part of the documentation, here](https://scores.readthedocs.io/en/latest/tutorials/Tutorial_Gallery.html). 
 
 ## Overview
 Here is a **curated selection** of the metrics, tools and statistical tests included in `scores`:
@@ -15,19 +19,19 @@ Here is a **curated selection** of the metrics, tools and statistical tests incl
 |-----------------------	|-----------------	|--------------	|
 | **[Continuous](https://scores.readthedocs.io/en/latest/included.html#continuous)**        	|Scores for evaluating single-valued continuous forecasts.                  	|Mean Absolute Error (MAE), Mean Squared Error (MSE), Root Mean Squared Error (RMSE), Additive Bias, Multiplicative Bias, Pearson's Correlation Coefficient, Flip-Flop Index, Quantile loss, Murphy score.              	|
 | **[Probability](https://scores.readthedocs.io/en/latest/included.html#probability)**       	|Scores for evaluating forecasts that are expressed as predictive distributions, ensembles, and probabilities of binary events.                 	|Brier Score, Continuous Ranked Probability Score (CRPS) for Cumulative Density Function (CDF), Threshold weighted CRPS for CDF, CRPS for ensembles, Receiver Operating Characteristic (ROC), Isotonic Regression (reliability diagrams).              	|
-| **[Categorical](https://scores.readthedocs.io/en/latest/included.html#categorical)**       	|Scores for evaluating forecasts based on categories.                	|Probability of Detection (POD), False Alarm Rate (FAR), Probability of False Detection (POFD), Success Ratio, Accuracy, Peirce's Skill Score, Critical Success Index (CSI), Gilbert Skill Score, Heidke Skill Score, Odds Ratio, Odds Ratio Skill Score, F1 score, FIxed Risk Multicategorical (FIRM) Score.               	|
+| **[Categorical](https://scores.readthedocs.io/en/latest/included.html#categorical)**       	|Scores for evaluating forecasts based on categories.                	|Probability of Detection (POD), False Alarm Ratio (FAR), Probability of False Detection (POFD), Success Ratio, Accuracy, Peirce's Skill Score, Critical Success Index (CSI), Gilbert Skill Score, Heidke Skill Score, Odds Ratio, Odds Ratio Skill Score, F1 score, FIxed Risk Multicategorical (FIRM) Score.               	|
 | **[Statistical Tests](https://scores.readthedocs.io/en/latest/included.html#statistical-tests)** 	|Tools to conduct statistical tests and generate confidence intervals.                 	|Diebold Mariano.              	|
 | **[Processing Tools](https://scores.readthedocs.io/en/latest/included.html#processing-tools-for-preparing-data)**        	|Tools to pre-process data.                 	|Data matching, Discretisation, Cumulative Density Function Manipulation.              	|
 
 
 `scores` not only includes common scores (e.g. MAE, RMSE), it includes novel scores not commonly found elsewhere (e.g. FIRM, Flip-Flop Index), complex scores (e.g. threshold weighted CRPS), and statistical tests (such as the Diebold Mariano test). Additionally, it provides pre-processing tools for preparing data for scores in a variety of formats including cumulative distribution functions (CDF). `scores` provides its own implementations where relevant to avoid extensive dependencies.
 
-`scores` primarily supports xarray datatypes for Earth system data allowing it to work with NetCDF4, hdf5, Zarr and GRIB data sources among others. `scores` uses Dask for scaling and performance. Some metrics work with pandas and we will aim to expand this capability. 
+`scores` primarily supports xarray datatypes for Earth system data allowing it to work with NetCDF4, HDF5, Zarr and GRIB data sources among others. `scores` uses Dask for scaling and performance. Some metrics work with pandas and we will aim to expand this capability. 
 
 All of the scores and metrics in this package have undergone a thorough scientific review. Every score has a companion Jupyter Notebook tutorial that demonstrates its use in practice.
 
 ## Contributing
-To find out more about contributing, see our [contributor's guide](https://github.com/nci/scores/blob/main/docs/contributing.md).
+To find out more about contributing, see our [Contributing Guide](https://scores.readthedocs.io/en/latest/contributing.html).
 
 All interactions in discussions, issues, emails and code (e.g. merge requests, code comments) will be managed according to the expectations outlined in the [ code of conduct ](https://github.com/nci/scores/blob/main/CODE_OF_CONDUCT.md) and in accordance with all relevant laws and obligations. This project is an inclusive, respectful and open project with high standards for respectful behaviour and language. The code of conduct is the Contributor Covenant, adopted by over 40,000 open source projects. Any concerns will be dealt with fairly and respectfully, with the processes described in the code of conduct.
 

docs/conf.py

@@ -9,7 +9,7 @@
 
 project = "scores"
 copyright = "Licensed under Apache 2.0 - https://www.apache.org/licenses/LICENSE-2.0"
-release = "0.8.3"
+release = "0.8.4"
 
 version = __version__
 

docs/contributing.md

@@ -4,7 +4,7 @@ Thank you for considering contributing to `scores`. Contributions of all kinds a
 
 These guidelines describe how to collaborate effectively.
 
-Types of contributions include bug reports, feature requests and pull requests. Contributions which are in line with the roadmap will be prioritised. The roadmap outlines the intentions for this package.
+Types of contributions include bug reports, feature requests, feedback and pull requests. Contributions which are in line with the roadmap will be prioritised. The roadmap outlines the intentions for this package.
 
 ## Roadmap
 - Addition of more scores, metrics and statistical techniques
@@ -13,9 +13,13 @@ Types of contributions include bug reports, feature requests and pull requests.
 - Increased support for machine learning library integration
 - Additional notebooks exploring complex use cases in depth
 
-## Bug Reports and Feature Requests
+## Code of Conduct and Respectful Behaviour
 
-Please submit bug reports and feature requests as issues in GitHub: [https://github.com/nci/scores/issues](https://github.com/nci/scores/issues).
+All interactions in discussions, issues, emails and code (e.g. pull requests, code comments) will be managed according to the expectations outlined in the [code of conduct](https://github.com/nci/scores/blob/main/CODE_OF_CONDUCT.md) and in accordance with all relevant laws and obligations. This project is an inclusive, respectful and open project with high standards for respectful behaviour and language. The code of conduct is the Contributor Covenant, adopted by over 40,000 open source projects. Any concerns will be dealt with fairly and respectfully, with the processes described in the code of conduct.
+
+## Bug Reports, Feature Requests and Feedback
+
+Please submit bug reports, feature requests and feedback as issues in GitHub: [https://github.com/nci/scores/issues](https://github.com/nci/scores/issues).
 
 ## Handling Security Concerns
 

docs/included.md

@@ -272,13 +272,13 @@
   - 
     [False alarm ratio (WWRP/WGNE Joint Working Group on Forecast Verification Research)](https://www.cawcr.gov.au/projects/verification/#FAR)
 * -  
-    - Fraction Correct, *see Accuracy*
+    - Fraction Correct (Accuracy)
   -
-    —
+    [API](https://scores.readthedocs.io/en/latest/api.html#scores.categorical.BasicContingencyManager.fraction_correct)
   - 
-    —
+    [Tutorial](https://scores.readthedocs.io/en/latest/tutorials/Binary_Contingency_Scores.html)
   - 
-    —
+    [Accuracy (WWRP/WGNE Joint Working Group on Forecast Verification Research)](https://www.cawcr.gov.au/projects/verification/#ACC)
 * -  
     - Frequency Bias (Bias Score)
   -
@@ -408,13 +408,13 @@
   - 
     [Threat score (WWRP/WGNE Joint Working Group on Forecast Verification Research)](https://www.cawcr.gov.au/projects/verification/#CSI)    
 * -  
-    - True Negative Rate, *see Specificity*
+    - True Negative Rate (Specificity)
   -
-    —
+    [API](https://scores.readthedocs.io/en/latest/api.html#scores.categorical.BasicContingencyManager.true_negative_rate)
   - 
-    —
+    [Tutorial](https://scores.readthedocs.io/en/latest/tutorials/Binary_Contingency_Scores.html)
   - 
-    —
+    [Wikipedia](https://en.wikipedia.org/wiki/Sensitivity_and_specificity)
 * -  
     - True Positive Rate (Hit Rate, Probability of Detection (POD), Sensitivity, Recall)
   -

docs/index.md

@@ -17,7 +17,7 @@ included
 api
 contributing
 data
-maintainer
-tutorials/Explanation
+tutorials/Tutorial_Gallery
 related_works
+maintainer
 ```

docs/maintainer.md

@@ -76,4 +76,45 @@ Information relevant for package maintenance
 |  README                  |  a new is score added    | in case it deserves a mention
 |  api.md                  |  a new function is added | each function must be added individually 
 |  included.md             |  a new function is added | each function (and each variation of the function name) must be added individually
-|  Explanation.ipynb       |  a new tutorial is added | navigation throughout the docs
+|  Tutorial_Gallery.ipynb  |  a new tutorial is added | navigation throughout the docs
+
+## This section covers checking the documentation renders properly in readthedocs
+
+### What documentation needs checking in readthedocs
+
+Each time an existing function is modified or a new function is added to `scores`, the rendering in readthedocs for any modified or newly created documentation must be checked. 
+
+This applies to each of the following documents:
+
+  - included.md
+  - API Documentation
+  - Tutorials (see also [tutorial rendering](#Tutorial-rendering) further below)
+  - (If applicable) README
+
+### Common rendering issues in readthedocs
+
+Frequent issues include:
+
+- Lists (including lists that use bullets, dot points, hyphens, numbers, letters etc.)
+  - Check **each** list appears and renders properly
+  - Check **all** indented lists/sub-lists for proper indentation
+- Figures: check **each** figure appears and renders properly
+- Plots: check **each** plot appears and renders properly
+- Tables: check **each** table appears and renders properly
+- Formulae: check **each** formula appears and renders properly
+- API Documentation: in addition to checking the above items, also confirm "Returns" and "Return Type" are rendering as expected
+
+### Tutorial rendering
+
+Things that render well in JupyterLab do not always render properly in readthedocs. Additionally, fixes that work well when built locally, don't always work when merged into the codebase. 
+
+To check the rendering of tutorials in readthedocs:
+  - Compare the tutorial in readthedocs against a version running in JupyterLab (as not everything renders in GitHub).
+  - Check the entirety of the tutorial (sometimes things will render properly in one section, while not rendering properly in a different section of the same tutorial).
+  - If you make any changes to the code cells, re-execute the Notebook in JupyterLab before committing, otherwise some things (e.g. some plots) won't render in readthedocs. Then re-check the tutorial in readthedocs to ensure the tutorial is still rendering properly.
+
+Ideally, also check the tutorial renders properly in nbviewer (there is a link at the top of each tutorial page in readthedocs).
+
+
+
+

src/scores/__init__.py

@@ -13,7 +13,7 @@
 import scores.sample_data
 import scores.stats.statistical_tests  # noqa: F401
 
-__version__ = "0.8.3"
+__version__ = "0.8.4"
 
 __all__ = [
     "scores.categorical",