prefer wheel-provided libraries, use RTLD_LOCAL

[jameslamb]

Description

Contributes to rapidsai/build-planning#118

Modifies libucx.load_library() in the following ways:

• prefer wheel-provided shared libraries to system installation
• expose environment variable RAPIDS_LIBUCX_PREFER_SYSTEM_LIBRARY for switching that preference
• load libraries with RTLD_LOCAL, to prevent adding symbols to the global namespace (dlopen docs)

Also updates all the pre-commit hook versions while I'm touching this repo. That was harmless, and I especially wanted to get up to the latest version of the RAPIDS copyright hook.

Notes for Reviewers

Version changes?

Proposing starting with 1.15.0.post2 because it's the oldest version supported at build and runtime by ucxx (code link).

And doing the following, in this order:

1. merge this PR
2. in a ucxx PR, test with these packages
3. publish 1.15.0.post2 packages (via manually triggering CI run here)
4. put up PRs to publish each of the other versions (https://pypi.org/project/libucx-cu12/#history)
   • 1.14.1.post2
   • 1.16.0.post2
   • 1.17.0.post1 (there was never a 1.17.0.post1)

[pentschev]

1.14.0.post2

Why is this needed? UCXX and UCX-Py both currently require at least UCX 1.15. Are there other use cases where this is important?

As to other changes it looks fine to me, I'll not pretend I understand most of it, but the overall idea seem reasonable to me.

[jameslamb]

@pentschev for why I'm proposing rebuilding 1.14 even though RAPIDS doesn't support it... same as the last time we talked about that: #6 (comment)

It's not that much incremental effort, and it ensures all the wheels work the same way regardless of which version you get. I don't feel too strongly about that, just proposing it because it's what we did the last time we had an update like this.

[pentschev]

for why I'm proposing rebuilding 1.14 even though RAPIDS doesn't support it... same as the last time we talked about that: #6 (comment)

It's not that much incremental effort, and it ensures all the wheels work the same way regardless of which version you get. I don't feel too strongly about that, just proposing it because it's what we did the last time we had an update like this.

I see, I would say I understand but I don't know if it's a great idea to keep on changing older versions. For instance, we may break things we now don't test anymore and that were previously working when we tested them. Similar to what we do with RAPIDS, I think it's probably ok to accept the past state of potentially broken libraries. IOW, I think we're fine doing it this once but I'm less convinced we should keep on doing that in the future, open to divergent opinions though.

[vyasr]

I'm inclined to agree with Peter here. After this change I suggest that we stop pushing fixes for older versions than we support in RAPIDS until someone specifically asks for them.

[vyasr]

@trxcllnt if everything looks fine to you here feel free to merge.

[jameslamb]

Thanks!

@vyasr I can handle repeating this for the other versions:

• 1.14.1.post2
• 1.16.0.post2
• 1.17.0.post1 (there was never a 1.17.0.post1)

[jameslamb]

Tested with a ucxx PR and confirmed that its CI passes with these changes: rapidsai/ucxx#334 (comment)

So I think we're good to proceed.