Online anomaly detection

[pranab]

Hi all,
I have a GitHub repo for anomaly detection with Spark/batch/scala implementation for various algorithms for univariate, multivariate and time series data. I am proposing online incremental reimplementation of them. It should be possible for most of the algorithms. It will be easy for all the stats based algorithms using running stat and some window. Your thoughts, comments are welcome. I already communicated with Max. He recommended posting in river/discussion. Here is the link to my repo. The project home page has links to blogs I wrote on various implementations

https://github.com/pranab/beymani

Pranab
https://www.linkedin.com/in/pkghosh/

[MaxHalford]

Hello @pranab! I suggest starting simple with a single method. Also, in general, we favor quality over quantity. Our goal isn't to implement every method out there, but instead to have those that generally considered to be battle-tested and popular. What would be a first model you might have in mind?

[pranab]

My understanding is that feature input is a dictionary. So it should work for 1 feature (univariate) as well

[pranab]

Running stat (mean and variance) is required for zscore based anomaly detection. In Mean Var class constructors the current stat values are set to zero. I need to make a change so that stat values from the previous run can be passed to the constructor as arguments for the next run. update() will use them.

[smastelini]

Hi @pranab. You can rely on the _from_state class method to achieve what you want:

river/river/stats/var.py

Lines 67 to 73 in 3732f70

	def _from_state(cls, n, m, sig, *, ddof=1):
	new = cls(ddof=ddof)
	new.mean = mean.Mean._from_state(n, m) # noqa
	# scale the second order statistic
	new._S = (n - ddof) * sig
	return new

I also had to do a similar action in the past and relied on them :)

[pranab]

Hi @smastelini thanks. RollingVar and RollingMean don't have those methods. I will add them

[pranab]

Hi @MaxHalford I created a pull request for zscore based anomaly detection

[MaxHalford]

Cool! I'm on holiday right until the 26th, I'll check it out soon :)

[pranab]

@MaxHalford not sure what do with the 2 errors under Action from my pull request

[MaxHalford]

Answered!