Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework how C++ wheels are handled in devcontainer builds #119

Open
vyasr opened this issue Nov 16, 2024 · 1 comment
Open

Rework how C++ wheels are handled in devcontainer builds #119

vyasr opened this issue Nov 16, 2024 · 1 comment

Comments

@vyasr
Copy link
Contributor

vyasr commented Nov 16, 2024

Currently C++ wheels are part of the devcontainer manifest and are therefore among the libraries built in those environments. However, in these environments the wheel build is designed to be a no-op because the library is already installed. The implementation involves 1) the CMake for the C++ exiting early if a find_package call finds that the library has already been built elsewhere, 2) the Python logic for load_library allowing through various cases where no library is found under the assumption that the Python extension modules will have suitable RPATHs baked in to find libraries, and 3) various dependency shenanigans to ensure that we get the desired dependency lists in different cases (e.g. DLFW vs. pip devcontainers vs. conda devcontainers). These changes require us to jump through a number of hoops downstream because they force us to consider cases where a C++ wheel package may be installed but actually not contain any DSO. With the benefit of some experience, it's time that we reimagine how we handle these various cases.

Rather than doing what we currently do, we can also satisfy all of these requirements by simply not installing the C++ wheels at all into environments where the system library is always preferred. The difficulty with accomplishing this so far in devcontainers and DLFW has been that this conflicts with the declared dependency lists, which state that e.g. pylibcudf depends on libcudf. Trying to remove the libcudf installation results in pip thinking the dependency is unsatisfied and going to install it. We can fix this by changing our dependencies.yaml files to ensure that when building with certain settings (config-settings for RBB that are passed to dfg's dependency generator) the C++ wheels are filtered out of the dependency list altogether. This solution should ensure that we never install C++ wheels except in environments where they are needed.

As a consequence of this, we will always have C++ wheels that contain DSOs (no more empty shells) and they can always default to loading the library from the wheel. They can also be consistently configured to allow loading system libraries (e.g. using an environment variable).

@vyasr
Copy link
Contributor Author

vyasr commented Nov 16, 2024

@trxcllnt pointed out that we could just reuse the use_cuda_wheels flag since that already pretty much covers this (we may want to rename it to capture the more general nature though).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant