River metrics evaluations

[giturra]

Hi everyone!

I have been studying this paper about word vectors. In the page 14-15, the authors talk about when evaluating stream classifiers is how to aggregate the values of the evaluation metric over time.

They consider three aggregation approaches, all of them implemented in MOA,

• Simple evaluator.
• Fading Factor Evaluator
• Reseting Evaluator

I was cheking out the evaluation metrics of River package, especially Accuracy and Kappa, and I was wordering if that approaches are implemented in River package.

Thanks :)

[MaxHalford]

Hey there! The simple evaluator seems to be just a global average, so you're good to go if you're using model_selection.progressive_val_score. As for the other methods, we don't support them in River. They look a bit specific, so at first it doesn't seem likely we would implement them into River. You would have to implement them yourself. However, I could imagine us implementing the "fading factor evaluator". It looks like it's just an exponential moving average.

[giturra]

Hey there! Thank you for your reply.

To be sure, If I use the model_selection.progressive_val_score, I'll get something similar to the simple evaluator? In that case, the package metrics evaluations are identical to those of scikit-learn metrics?

Thank :)

[MaxHalford]

To be sure, If I use the model_selection.progressive_val_score, I'll get something similar to the simple evaluator? In that case, the package metrics evaluations are identical to those of scikit-learn metrics?

Yes and yes!

[giturra]

okay! Thank you @MaxHalford :)