how to scale rrcf to detect thousands of time series

[zcsh]

dear mdb,
I have tried using rrcf to detect anomalies in a single time series, and konwn it is efficient in doing this work. But how can I build an application with rrcf to detect several thousands of time series that stream in every 1 minute synchronously with reasonable cpu and memory consumption? Have you any suggestions?

[mdbartos]

Here are a few suggestions:

• Instead of shingling, I would recommend computing summary statistics that capture the type of anomaly you are looking for. This will reduce the dimension of the points you are inserting into each tree, which will in turn result in better performance. So, if you are looking for spikes you may your points to consist of second central differences. If you are looking for long-term trends, you may want your points to consist of rolling means at different window sizes, etc.

• If data is arriving too quickly to be inserted, you can compute a rolling summary statistic over some buffered input (mean, median, max, etc. or some combination of these). This would reduce the number of points that need to be inserted.

• Use parallelization, as discussed in this thread: insert_point is slow #28

• If the time series are independent, you could run different rrcf instances for each time series, using separate processes or server instances.