Benchmarking module

[fredmontet]

Loading data

• some basic datasets
• custom dataset
• TimeEval format (https://timeeval.readthedocs.io/en/latest/)

Model training

• encapsulation of models to be tested in a dedicated object (e.g. ModelToBenchmark )
• possibility of running the benchmark on a series of datasets
• optimization with Optuna using e.g. a boolean #81

Comparison for Predictors

• Test of multiple predictors and select the best one (check prediction quality)
• Performance plot for each model and dataset
• Comparison of characteristics of selected models (confidence interval, point prediction or not, etc.)
• Test of multiple prediction windows and select the best one #82
• Test at different time along the tested window

Comparison for Detectors

• Test of multiple statistical detection thresholds and select the best one

[fredmontet]

1. Chargement des donnée
2. POC de module de benchmark (MVP test de 2 modèle, 2 datasets)
   MS1 - Validation archi
3. Ajout de features (optimisation, longueur de prediction, features de modèle)
   MSx - feature x
4. Reporting et format des résultats (.md, .png, .pdf, exécution hors de notebooks de préférence)

[fredmontet]

Issue is partially completed, the checkboxes have been reviews. Issue will be closed because it is not so relevant anymore.