How to use ensembles with anomalies detectors

[dberardo-com]

I would like to setup an anomalies detector for monitoring single spectral amplitudes / frequency bands of a signal transformed using SDFT.

The idea is to have one input time series and then a bucketing function for the amplitude/frequency bands. For each one of these bands i would like to set a GaussianScorer/ThresholdFilter that would monitor that specific band's distribution and return 1 if that frequency is within a "dangerous" region.

The ensemble will then be used to aggregate the output of the single scorer and return an alarm signal based on an aggregation of the single ThresholdFilter's signals (perhaps just a fix threshold, like if 50% of the scorer return 1 then emits a signal).

I thought of using ensembles for this. Is this a possibility?

Link related discussions:

• Timeseries forecast with irregular input streams #971

[dberardo-com]

So all in all the problem can be conceptually be boiled down to: "how to detect anomalies not on a single value, but on an histogram of values" and the ensemble of scorer can be thought of as a BoxPlot on the different buckets of the histogram.

[MaxHalford]

Hey there. I'm not sure I understand everything. Say you want to have 5 anomaly detectors, are you looking for some way to merge the scores from all 5 anomaly detectors into a single score?

[dberardo-com]

exactly, is it possible via ensembles?

[MaxHalford]

If you're talking about the stuff in the ensemble, the answer is sadly "no, sorry :(". That stuff is meant to be used for supervised learning.

For anomaly detection, my instinct tells me that we should have some a simple aggregation method that allows using any formula: sum, statistics.mean, max, etc. What do you think? I can implement it very soon.

[dberardo-com]

it would be nice to use a pipeline or a wrapper to achieve this. ideally like a ThresholdFilter around the 5 detectors. Otherwise a statistical aggregator could also do the job (best in a pipeline as well).

[dberardo-com]

Also as a side question, is it possible to break out of a pipeline if a condition is met?

like in pipeline A-B-C if block A returns None then the blocks B and C are never executed (circuit breaker)

[dberardo-com]

i think i did see this comment somewhere else but wanted to make sure. ok then, i will probably go for an exception.

this is by the way, all still related to the TumblingWindow feature that I was telling you about in another issue. Since I want to use SDFT to perform anomalies detection, I need the Tumbling block to make sure that the incoming samples are all equally spaced. But then i also need to make an interpolation in the TumblingWindow in order to fill missing data, otherwise SDFT will probably complain (i could not find a way in the internet to work with missing values in DFT other than reconstructing them). However, i would like interpolated values not to be considered in further steps of the pipeline, but only in the SDFT block where they are needed "artficially".

I hope this makes sense, this is the reason for my question

[dberardo-com]

also FYI, this is what the fourier spectrum on "daily" periods on the WaterFlow dataset looks like:

as you can see from the heatmap, the spectrum frequency "bins" can be used to detect anomalies, but i would need on detector per frequency bin.

P.S. the x axis of the heatmap are daily periods and the y axis are frequency bins in the spectrum. the value/color of the heatmap is the amplitute of that harmonic band in the spectrum

[MaxHalford]

You're making a lot of sense. What you're doing is simply very specific to your task. And what you're getting to is essentially a feature request. I will definitely keep what you wrote in mind and translate into features at some point. In particular I think there's a lot of value in doing anomaly detection based on Fourier amplitudes.

[dberardo-com]

You're making a lot of sense.

i take this as a positive comment, never heard it in the past :D

anyway this message was just to motivate further this discussion, which is now linked to this PR: #971

What you're doing is simply very specific to your task.

do you think so? i mean, having multiple detectors in parallel and aggregating their results in a second stage sounds to me pretty sensible. Btw i got inspired by this article: https://medium.com/low-code-for-advanced-data-science/anomaly-detection-for-predictive-maintenance-control-charts-abcbe656c2d0

where they also have a first level alert (single detector alert) and a second level alert (multiple detectors score an anomaly).

[MaxHalford]

do you think so? i mean, having multiple detectors in parallel and aggregating their results in a second stage sounds to me pretty sensible.

Oh totally. I meant it's difficult to understand the intricacies of your task. I'll add a card for ensembling anomaly detection scores to our roadmap :)