Merge branch 'scikit-learn-contrib:master' into master

.bumpversion.cfg

@@ -14,3 +14,8 @@ replace = __version__ = "{new_version}"
 [bumpversion:file:doc/conf.py]
 search = version = "{current_version}"
 replace = version = "{new_version}"
+
+[bumpversion:file:CITATION.cff]
+search = version = "{current_version}"
+replace = version = "{new_version}"
+

.github/workflows/test.yml

@@ -33,7 +33,7 @@ jobs:
             python-version: "3.11"
             numpy-version: 1.25.2
           - os: macos-latest
-            python-version: "3.11"
+            python-version: "3.10"
             numpy-version: 1.25.2
     defaults:
       run:
@@ -48,6 +48,7 @@ jobs:
           python-version: ${{ matrix.python-version }}
           environment-file: environment.ci.yml
           channels: defaults, conda-forge
+          miniconda-version: "latest"
       - name: Install numpy
         run: conda install numpy=${{ matrix.numpy-version }}
       - name: Check linting

CITATION.cff

@@ -0,0 +1,48 @@
+cff-version: 1.2.0
+message: "If you use this software, please cite it as below."
+authors:
+- family-names: "Cordier"
+  given-names: "Thibault"
+  orcid: "https://orcid.org/0000-0000-0000-0000"
+title: "MAPIE - Model Agnostic Prediction Interval Estimator"
+version: 0.8.3
+date-released: 2019-04-30
+url: "https://github.com/scikit-learn-contrib/MAPIE"
+preferred-citation:
+  type: conference-paper
+  title: "Flexible and Systematic Uncertainty Estimation with Conformal Prediction via the MAPIE library"
+  authors:
+  - family-names: "Cordier"
+    given-names: "Thibault"
+  - family-names: "Blot"
+    given-names: "Vincent"
+  - family-names: "Lacombe"
+    given-names: "Louis"
+  - family-names: "Morzadec"
+    given-names: "Thomas"
+  - family-names: "Capitaine"
+    given-names: "Arnaud"
+  - family-names: "Brunel"
+    given-names: "Nicolas"
+  collection-title: "Conformal and Probabilistic Prediction with Applications"
+  collection-type: proceedings
+  pages: "549--581"
+  year: 2023
+  organization: "PMLR"
+old-citation:
+  type: article
+  authors:
+  - family-names: "Taquet"
+    given-names: "Vianney"
+  - family-names: "Blot"
+    given-names: "Vincent"
+  - family-names: "Morzadec"
+    given-names: "Thomas"
+  - family-names: "Lacombe"
+    given-names: "Louis"
+  - family-names: "Brunel"
+    given-names: "Nicolas"
+  doi: "10.48550/arXiv.2207.12274"
+  journal: "arXiv preprint arXiv:2207.12274"
+  title: "MAPIE: an open-source library for distribution-free uncertainty quantification"
+  year: 2021

CODE_OF_CONDUCT.md

@@ -60,7 +60,7 @@ representative at an online or offline event.
 
 Instances of abusive, harassing, or otherwise unacceptable behavior may be
 reported to the community leaders responsible for enforcement at
-tcordier@quantmetry.com.
+thibault.a.cordier@capgemini.com.
 All complaints will be reviewed and investigated promptly and fairly.
 
 All community leaders are obligated to respect the privacy and security of the

HISTORY.rst

@@ -2,6 +2,16 @@
 History
 =======
 
+0.8.3 (2024-**-**)
+------------------
+
+* Fix conda versionning.
+* Reduce precision for test in `MapieCalibrator`.
+* Fix invalid certificate when downloading data.
+* Add citations utility to the documentation.
+* Add documentation for metrics.
+* Add explanation and example for symmetry argument in CQR.
+
 0.8.3 (2024-03-01)
 ------------------
 
@@ -14,7 +24,7 @@ History
 0.8.2 (2024-01-11)
 ------------------
 
-* * Resolve issue still present in 0.8.1 by updating pandas.
+* Resolve issue still present in 0.8.1 by updating pandas.
 
 0.8.1 (2024-01-11)
 ------------------

README.rst

@@ -1,38 +1,39 @@
 .. -*- mode: rst -*-
 
-|GitHubActions|_ |Codecov|_ |ReadTheDocs|_ |License|_ |PythonVersion|_ |PyPi|_ |Conda|_ |Release|_ |Commits|_ |DOI|_
+|GitHubActions| |Codecov| |ReadTheDocs| |License| |PythonVersion| |PyPi| |Conda| |Release| |Commits| |DOI|
 
 .. |GitHubActions| image:: https://github.com/scikit-learn-contrib/MAPIE/actions/workflows/test.yml/badge.svg
-.. _GitHubActions: https://github.com/scikit-learn-contrib/MAPIE/actions
+   :target: https://github.com/scikit-learn-contrib/MAPIE/actions
 
 .. |Codecov| image:: https://codecov.io/gh/scikit-learn-contrib/MAPIE/branch/master/graph/badge.svg?token=F2S6KYH4V1
-.. _Codecov: https://codecov.io/gh/scikit-learn-contrib/MAPIE
+   :target: https://codecov.io/gh/scikit-learn-contrib/MAPIE
 
-.. |ReadTheDocs| image:: https://readthedocs.org/projects/mapie/badge
-.. _ReadTheDocs: https://mapie.readthedocs.io/en/latest
+.. |ReadTheDocs| image:: https://readthedocs.org/projects/mapie/badge/?version=stable
+    :target: https://mapie.readthedocs.io/en/stable/?badge=stable
+    :alt: Documentation Status
 
-.. |License| image:: https://img.shields.io/github/license/simai-ml/MAPIE
-.. _License: https://github.com/scikit-learn-contrib/MAPIE/blob/master/LICENSE
+.. |License| image:: https://img.shields.io/github/license/scikit-learn-contrib/MAPIE
+   :target: https://github.com/scikit-learn-contrib/MAPIE/blob/master/LICENSE
 
 .. |PythonVersion| image:: https://img.shields.io/pypi/pyversions/mapie
-.. _PythonVersion: https://pypi.org/project/mapie/
+   :target: https://pypi.org/project/mapie/
 
 .. |PyPi| image:: https://img.shields.io/pypi/v/mapie
-.. _PyPi: https://pypi.org/project/mapie/
+   :target: https://pypi.org/project/mapie/
 
 .. |Conda| image:: https://img.shields.io/conda/vn/conda-forge/mapie
-.. _Conda: https://anaconda.org/conda-forge/mapie
+   :target: https://anaconda.org/conda-forge/mapie
 
 .. |Release| image:: https://img.shields.io/github/v/release/scikit-learn-contrib/mapie
-.. _Release: https://github.com/scikit-learn-contrib/MAPIE/releases
+   :target: https://github.com/scikit-learn-contrib/MAPIE/releases
 
 .. |Commits| image:: https://img.shields.io/github/commits-since/scikit-learn-contrib/mapie/latest/master
-.. _Commits: https://github.com/scikit-learn-contrib/MAPIE/commits/master
+   :target: https://github.com/scikit-learn-contrib/MAPIE/commits/master
 
 .. |DOI| image:: https://img.shields.io/badge/10.48550/arXiv.2207.12274-B31B1B.svg
-.. _DOI: https://arxiv.org/abs/2207.12274
+   :target: https://arxiv.org/abs/2207.12274
 
-.. image:: https://github.com/simai-ml/MAPIE/raw/master/doc/images/mapie_logo_nobg_cut.png
+.. image:: https://github.com/scikit-learn-contrib/MAPIE/raw/master/doc/images/mapie_logo_nobg_cut.png
     :width: 400
     :align: center
 
@@ -157,7 +158,7 @@ The full documentation can be found `on this link <https://mapie.readthedocs.io/
 ===============
 
 You are welcome to propose and contribute new ideas.
-We encourage you to `open an issue <https://github.com/simai-ml/MAPIE/issues>`_ so that we can align on the work to be done.
+We encourage you to `open an issue <https://github.com/scikit-learn-contrib/MAPIE/issues>`_ so that we can align on the work to be done.
 It is generally a good idea to have a quick discussion before opening a pull request that is potentially out-of-scope.
 For more information on the contribution process, please go `here <CONTRIBUTING.rst>`_.
 
@@ -168,27 +169,32 @@ For more information on the contribution process, please go `here <CONTRIBUTING.
 MAPIE has been developed through a collaboration between Quantmetry, Michelin, ENS Paris-Saclay,
 and with the financial support from Région Ile de France and Confiance.ai.
 
-|Quantmetry|_ |Michelin|_ |ENS|_ |Confiance.ai|_  |IledeFrance|_ 
+|Quantmetry| |Michelin| |ENS| |Confiance.ai| |IledeFrance|
 
 .. |Quantmetry| image:: https://www.quantmetry.com/wp-content/uploads/2020/08/08-Logo-quant-Texte-noir.svg
-    :width: 150
-.. _Quantmetry: https://www.quantmetry.com/
+    :height: 35px
+    :width: 140px
+    :target: https://www.quantmetry.com/
 
-.. |Michelin| image:: https://www.michelin.com/wp-content/themes/michelin/public/img/michelin-logo-en.svg
-    :width: 100
-.. _Michelin: https://www.michelin.com/en/
+.. |Michelin| image:: https://agngnconpm.cloudimg.io/v7/https://dgaddcosprod.blob.core.windows.net/corporate-production/attachments/cls05tqdd9e0o0tkdghwi9m7n-clooe1x0c3k3x0tlu4cxi6dpn-bibendum-salut.full.png
+    :height: 50px
+    :width: 45px
+    :target: https://www.michelin.com/en/
 
 .. |ENS| image:: https://file.diplomeo-static.com/file/00/00/01/34/13434.svg
-    :width: 100
-.. _ENS: https://ens-paris-saclay.fr/en
+    :height: 35px
+    :width: 140px
+    :target: https://ens-paris-saclay.fr/en
 
 .. |Confiance.ai| image:: https://pbs.twimg.com/profile_images/1443838558549258264/EvWlv1Vq_400x400.jpg
-    :width: 100
-.. _Confiance.ai: https://www.confiance.ai/
+    :height: 45px
+    :width: 45px
+    :target: https://www.confiance.ai/
 
-.. |IledeFrance| image:: https://www.iledefrance.fr/themes/custom/portail_idf/logo.svg
-    :width: 100
-.. _IledeFrance: https://www.iledefrance.fr/
+.. |IledeFrance| image:: https://www.iledefrance.fr/sites/default/files/logo/2024-02/logoGagnerok.svg
+    :height: 35px
+    :width: 140px
+    :target: https://www.iledefrance.fr/
 
 
 🔍  References
@@ -222,4 +228,10 @@ and with the financial support from Région Ile de France and Confiance.ai.
 📝 License
 ==========
 
-MAPIE is free and open-source software licensed under the `3-clause BSD license <https://github.com/simai-ml/MAPIE/blob/master/LICENSE>`_.
+MAPIE is free and open-source software licensed under the `3-clause BSD license <LICENSE>`_.
+
+
+📚 Citation
+===========
+
+If you use MAPIE in your research, please cite using `citations file <CITATION.cff>`_ on our repository.

doc/index.rst

@@ -58,6 +58,13 @@
    examples_calibration/index
    notebooks_calibration
 
+.. toctree::
+   :maxdepth: 2
+   :hidden:
+   :caption: METRICS
+
+   theoretical_description_metrics
+
 .. toctree::
    :maxdepth: 2
    :hidden:

doc/notebooks_classification.rst

@@ -6,8 +6,8 @@ problems for computer vision settings that are too heavy to be included in the e
 galleries.
 
 
-1. Estimating prediction sets on the Cifar10 dataset : `notebook <https://github.com/scikit-learn-contrib/MAPIE/tree/master/notebooks/classification/Cifar10.ipynb>`_
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
+1. Estimating prediction sets on the Cifar10 dataset : `cifar_notebook <https://github.com/scikit-learn-contrib/MAPIE/tree/master/notebooks/classification/Cifar10.ipynb>`_
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-2. Top-label calibration for outputs of ML models : `notebook <https://github.com/scikit-learn-contrib/MAPIE/tree/master/notebooks/classification/top_label_calibration.ipynb>`_
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+2. Top-label calibration for outputs of ML models : `top_label_notebook <https://github.com/scikit-learn-contrib/MAPIE/tree/master/notebooks/classification/top_label_calibration.ipynb>`_
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

doc/notebooks_multilabel_classification.rst

@@ -5,8 +5,8 @@ The following examples present advanced analyses
 on multi-label classification problems with different 
 methods proposed in MAPIE.
 
-1. Overview of Recall Control for Multi-Label Classification : `notebook <https://github.com/scikit-learn-contrib/MAPIE/tree/master/notebooks/classification/tutorial_multilabel_classification_recall.ipynb>`_
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+1. Overview of Recall Control for Multi-Label Classification : `recall_notebook <https://github.com/scikit-learn-contrib/MAPIE/tree/master/notebooks/classification/tutorial_multilabel_classification_recall.ipynb>`_
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-2. Overview of Precision Control for Multi-Label Classification : `notebook <https://github.com/scikit-learn-contrib/MAPIE/tree/master/notebooks/classification/tutorial_multilabel_classification_precision.ipynb>`_
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+2. Overview of Precision Control for Multi-Label Classification : `precision_notebook <https://github.com/scikit-learn-contrib/MAPIE/tree/master/notebooks/classification/tutorial_multilabel_classification_precision.ipynb>`_
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

doc/notebooks_regression.rst

@@ -8,11 +8,11 @@ This section lists a series of Jupyter notebooks hosted on the MAPIE Github repo
 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
 
-2. Estimating the uncertainties in the exoplanet masses : `notebook <https://github.com/scikit-learn-contrib/MAPIE/tree/master/notebooks/regression/exoplanets.ipynb>`_
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+2. Estimating the uncertainties in the exoplanet masses : `exoplanet_notebook <https://github.com/scikit-learn-contrib/MAPIE/tree/master/notebooks/regression/exoplanets.ipynb>`_
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
 
-3. Estimating prediction intervals for time series forecast with EnbPI and ACI : `notebook <https://github.com/scikit-learn-contrib/MAPIE/tree/master/notebooks/regression/ts-changepoint.ipynb>`_
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+3. Estimating prediction intervals for time series forecast with EnbPI and ACI : `ts_notebook <https://github.com/scikit-learn-contrib/MAPIE/tree/master/notebooks/regression/ts-changepoint.ipynb>`_
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
 

doc/quick_start.rst

@@ -7,11 +7,9 @@ In regression settings, **MAPIE** provides prediction intervals on single-output
 In classification settings, **MAPIE** provides prediction sets on multi-class data.
 In any case, **MAPIE** is compatible with any scikit-learn-compatible estimator.
 
-Estimate your prediction intervals
-==================================
 
 1. Download and install the module
-----------------------------------
+==================================
 
 Install via ``pip``:
 
@@ -33,7 +31,7 @@ To install directly from the github repository :
 
 
 2. Run MapieRegressor
----------------------
+=====================
 
 Let us start with a basic regression problem. 
 Here, we generate one-dimensional noisy data that we fit with a linear model.
@@ -114,8 +112,8 @@ It is given by the alpha parameter defined in ``MapieRegressor``, here equal to
 thus giving target coverages of ``0.95`` and ``0.68``.
 The effective coverage is the actual fraction of true labels lying in the prediction intervals.
 
-2. Run MapieClassifier
-----------------------
+3. Run MapieClassifier
+=======================
 
 Similarly, it's possible to do the same for a basic classification problem.
 

doc/theoretical_description_binary_classification.rst

@@ -1,18 +1,18 @@
-.. title:: Theoretical Description : contents
+.. title:: Theoretical Description Binary Classification : contents
 
 .. _theoretical_description_binay_classification:
 
-=======================
+#######################
 Theoretical Description
-=======================
+#######################
 
 There are mainly three different ways to handle uncertainty quantification in binary classification:
 calibration (see :doc:`theoretical_description_calibration`), confidence interval (CI) for the probability
 :math:`P(Y \vert \hat{\mu}(X))` and prediction sets (see :doc:`theoretical_description_classification`).
 These 3 notions are tightly related for score-based classifier, as it is shown in [1]. 
 
 Prediction sets can be computed in the same way for multiclass and binary classification with
-:class:`~mapie.calibration.MapieClassifier`, and there are the same theoretical guarantees.
+:class:`~mapie.classification.MapieClassifier`, and there are the same theoretical guarantees.
 Nevertheless, prediction sets are often much less informative in the binary case than in the multiclass case.
 
 From Gupta et al [1]:
@@ -83,8 +83,8 @@ for the labels of test objects which are guaranteed to be well-calibrated under
 that the observations are generated independently from the same distribution [2].
 
 
-4. References
--------------
+References
+----------
 
 [1] Gupta, Chirag, Aleksandr Podkopaev, and Aaditya Ramdas.
 "Distribution-free binary classification: prediction sets, confidence intervals, and calibration."