NumPy 2.0 compatibility

[agriyakhetarpal]

Description

This PR is an attempt to compile with NumPy v2 at build time plus relax the Cython<3 requirement, so that river wheels will be compatible with both NumPy v2 and NumPy v1.X. Please refer to #1630 for more context around putting up this pull request.

Additional context

xref: pyodide/pyodide#4925

[agriyakhetarpal]

The CI logs show that it still installs NumPy 1.26.4. I am not acquainted with using Poetry, especially not for compiled projects. How should I lift/make Poetry lift this bound (and for Cython)? pip install -e . locally does use NumPy v2 and Cython 3 to compile.

[gbolmier]

Thank you so much for all your efforts @agriyakhetarpal!

The CI logs show that it still installs NumPy 1.26.4. I am not acquainted with using Poetry, especially not for compiled projects. How should I lift/make Poetry lift this bound (and for Cython)? pip install -e . locally does use NumPy v2 and Cython 3 to compile.

Try incrementing this line

[agriyakhetarpal]

I don't think that helped, it's picking up 1.26 from somewhere... what are we missing?

[e10e3]

If you look at the updated lockfile, NumPy's version here is still at 1.26. This is what Poetry installs. For Poetry to select a higher version, you need to update Scikit-learn, because the version currently selected forbids NumPy 2. 😃

I believe this CI does not build wheels (if this is what you are trying to do), it only installs River and runs the tests.
You can build a wheel locally with pip wheel .. (I use Pip here because Poetry says it only supports pure Python wheels.)

[agriyakhetarpal]

If you look at the updated lockfile, NumPy's version here is still at 1.26. This is what Poetry installs. For Poetry to select a higher version, you need to update Scikit-learn, because the version currently selected forbids NumPy 2. 😃

Ah, thank you! That must be it – I'll bump the rest of the dependencies with this, update the lockfile, and check when the requirement gets relaxed fully.

I believe this CI does not build wheels (if this is what you are trying to do), it only installs River and runs the tests.
You can build a wheel locally with pip wheel .. (I use Pip here because Poetry says it only supports pure Python wheels.)

Building wheels or testing an editable/non-editable installation should be the same thing, since in any case the build backend is going to compile a wheel in a temporary location, install from it, and then discard it. I'll check if I need to do anything extra when I try building wheels. The local wheel build with both frontends: pip wheel and pypa/build succeeded (which used NumPy 2 because I added that constraint for the build-time requirement – or so I think unless poetry's backend did something weird :P)

[agriyakhetarpal]

Okay, we're at numpy 2.0.2 now, and here's where I'll start to expect seeing tests that fail and the builds passing. I'll check this later in the day.

[e10e3]

Building wheels or testing an editable/non-editable installation should be the same thing, since in any case the build backend is going to compile a wheel in a temporary location, install from it, and then discard it.

Ooh, I wasn't aware of how it works. Thank you 😄

[MaxHalford]

Thanks everyone for working on this.

For the failing tests: the philosophy behind River is to hide the fact that numpy is sometimes used under the hood. River objects are supposed to take as input and output Python objects. So we have to make some changes so that Python floats are returned, and not NumPy floats. I suppose this isn't very difficult to do, it's just a matter to of pin-pointing where the NumPy originates from.

[MaxHalford]

This PR fixes most of the unit tests.

@smastelini the remaining broken unit tests are related to the drift module. Would you have some time to take a look?

[smastelini]

Sure! I will take a look at it in the next couple days. Today I'm going to a conference and will spent the day on the road.

[MaxHalford]

Nice! Enjoy your conference :)

[agriyakhetarpal]

It looks like things are working without an issue, but from the logs, I still see Installing numpy (2.0.2) – I'd like to try if it's possible to bump it further to NumPy >= 2.1. It's coming from thinc, and in-turn, from spacy.

Another thing I noticed is that polars's versions 1.13.0 and 1.13.1 were yanked over the weekend, so I'll re-generate the lockfile to use 1.14.

[agriyakhetarpal]

Bumping to NumPy 2.1 is not possible right now because it constrains SciPy (which, as of 1.14.1, requires Python 3.10 and later, and river still supports Python 3.9) and because spaCy does not yet have a stable v4 release for its pre-releases on PyPI that relaxes NumPy to be <3. However, building against NumPy 2.0 is reasonable enough and should be fine here. Thanks for all the help! I'll mark this as ready for review now, with some additional comments to help facilitate the review.

[MaxHalford]

Thank you so much @agriyakhetarpal, and kudos on the self-review. I guess we can merge!

[agriyakhetarpal]

Thanks for the merge! Are there any additional items I can help with to address before a release with this PR is made?

[MaxHalford]

I guess @gbolmier (or you) can take a look at whether adding support for Python 3.13 is a low hanging fruit or not. If it is, we may as well include it in the next release too.

[agriyakhetarpal]

It doesn't look like it is easy to do, but I'll explore the package in more detail sometime and try to contribute here in other ways if I can, if I have time 😄

[MaxHalford]

Actually I think it's acceptable to only support NumPy v2 if that makes things easier!

[agriyakhetarpal]

Oh, do you mean supporting NumPy >=2 for both build time and runtime? That could work for River as an application, but River is being used as a library downstream in quite a few projects (https://www.wheelodex.org/projects/river/rdepends/) where I would think it's too early for an upper bound that's so high, isn't it?

[MaxHalford]

Well I would say these libraries can always pin River to a previous version if they want it to keep working. But I guess you're right.

[gbolmier]

Shout-out to @agriyakhetarpal for the great work he put in 🙌

I guess @gbolmier (or you) can take a look at whether adding support for Python 3.13 is a low hanging fruit or not. If it is, we may as well include it in the next release too.

I should have time to give it a try in the next days. I'll ask help from @AdilZouitine if there is non-trivial rust debugging to do.

Bumping to NumPy 2.1 is not possible right now because it constrains SciPy (which, as of 1.14.1, requires Python 3.10 and later, and river still supports Python 3.9)

I think it would make sense to not support 3.9 anymore like NumPy and SciPy do.