Mathematical Background of River

[usmanmalik57]

Hello Max Halford, Thankyou for this amazing library for incremental learning.

Could you please add mathematical background for the incremental learning approaches you are using in this library, in the documentation section.
For instance, if you talk about logistic regression, how do you update the coefficients when a new class is encountered? Are the older coefficients incrementally forgotten or their is some weight decay strategy. Could not find any help in documentation with regards to it.
Thanks!

[MaxHalford]

Hello there.

Hello Max Halford

There's a whole team of us, it's not just me :)

Could you please add mathematical background for the incremental learning approaches you are using in this library, in the documentation section.

We have a lot of models. I agree that we need to become better at explaining the inner workings of our methods. Thankfully, 90% of our work is based on well understood methods that are explained in papers and blog posts across the internet.

For instance, if you talk about logistic regression, how do you update the coefficients when a new class is encountered? Are the older coefficients incrementally forgotten or their is some weight decay strategy.

This questions is much too open and a bit misleading. Logistic regression is a binary classification model, so there aren't new classes that can appear. We have softmax regression, which is a multi-class classification model. It maintains a weight vector for each class, and thus initialises new weights when a new class appears. Then we compute a multi-class loss for each observation, which gives us gradients that we use to update each weight vector.

[usmanmalik57]

Thank you for your response.
I was talking about using logistic regression in combination with one-vs-rest classifier for multiclass classification. Could you redirect me to some of the papers that you have consulted for implementing learning. I mean there are plenty over the internet: progressive neural networks, distillation approaches, deep adaptation etc. Or you are simply, updating weight vector for softmax regression as soon as a new class arrives which is kind of distillation, am I right?

[MaxHalford]

We use essentially the same logic that scikit-learn does, so I recommend starting from their document on stochastic gradient descent.