Distributed Streaming? Merging Models?

[bubbajoe]

Hello,

I am new to this library and i really like what you guys are building here.

I would like to know if there is any way to use this library in a more distributed way?

For example:
I have X number of servers consuming data and I want to have X number of models being updated and periodically merged into 1 common model.

Any idea how I can achieve this or something similar?

Thanks!

[raphaelsty]

Hi @bubbajoe,

River is not developed for distributed computing. This means that it is not easy to implement but not impossible. If you have real problems with data ingestion you can eventually implement the Hogwild solution.

Hogwild!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent

[MaxHalford]

Hello @bubbajoe. Can I ask why you're asking for this? I have encountered a situation where the data is arriving too fast? Some more context will help us answer as this is a difficult question.

[kulbachcedric]

Hello @bubbajoe, hello @MaxHalford,

I think ray.io could do the job. In my case the decentralization of different river pipelines slowed down the learning process of an ensemble. After decentralizing a model, the question arises, how to merge the models into one common river model?