Make AnomalyDetectors compatible to tracks

[kulbachcedric]

Hi all,

currently the river.anomaly.AnomalyDetector is not compatible with river.evaluate.Track (or with the Metrics).
Based on scikit-learn one could make each AnomalyDetector algorithm also inherit from river.base.Classifer?
Whereby, the river.anomaly.AnomalyDetector only implements a learn_score_one function and the learn_one function is implemented from river.base.Classifier, where y is passed as None.

What do you think?

Best
Cedric

[MaxHalford]

It's an interesting topic.

All anomaly detectors output an anomaly score. In River this is through the score_one method. In scikit-learn it's the score_samples method. Scores can then be converted into decisions by thresholding the score. scikit-learn does that arbitrarily with the predict function. We don't do that in River. Instead we provide anomaly detector wrappers that control the threshold (see ConstantThresholder and QuantileThresholder). These will produce a score that is either 0 or 1 when score_one is called.

The scores of anomaly detectors can be used to update metrics. I wouldn't say there incompatible. What makes you feel that metrics don't work for anomaly detectors?

As for tracks, we indeed do not support anomaly detectors. But it shouldn't be too difficult to do. The same goes for testing: there are test suites for classification and regression, but not for other tasks such as anomaly detection, clustering, time series, etc.

[kulbachcedric]

Hi Max,

I saw the wrappers.
Sure, the scores of an AnomalyDetector can be used to update e.g. the ROCAUC metric, but the problem here is the works_with method of the metric. The ROCAUC metric is a metrics.base.ClassificationMetric which uses the utils.inspect.isclassifier method to check that the used model is an instance ob base.Classifier.
Since an AnomalyDetector is not an instance of base.Classifier, the method will return False.

The metric.works_with method is used within the evaluate._progressive_validation method, which returns a ValueError.

I'll try to make a suggestion.

[MaxHalford]

That's a good catch! My gut feeling is that we could extend the works_with method to allow for anomaly detectors. I would prefer that to having anomaly detectors inherit from base.Classifier.