You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently C++ wheels are part of the devcontainer manifest and are therefore among the libraries built in those environments. However, in these environments the wheel build is designed to be a no-op because the library is already installed. The implementation involves 1) the CMake for the C++ exiting early if a find_package call finds that the library has already been built elsewhere, 2) the Python logic for load_library allowing through various cases where no library is found under the assumption that the Python extension modules will have suitable RPATHs baked in to find libraries, and 3) various dependency shenanigans to ensure that we get the desired dependency lists in different cases (e.g. DLFW vs. pip devcontainers vs. conda devcontainers). These changes require us to jump through a number of hoops downstream because they force us to consider cases where a C++ wheel package may be installed but actually not contain any DSO. With the benefit of some experience, it's time that we reimagine how we handle these various cases.
Rather than doing what we currently do, we can also satisfy all of these requirements by simply not installing the C++ wheels at all into environments where the system library is always preferred. The difficulty with accomplishing this so far in devcontainers and DLFW has been that this conflicts with the declared dependency lists, which state that e.g. pylibcudf depends on libcudf. Trying to remove the libcudf installation results in pip thinking the dependency is unsatisfied and going to install it. We can fix this by changing our dependencies.yaml files to ensure that when building with certain settings (config-settings for RBB that are passed to dfg's dependency generator) the C++ wheels are filtered out of the dependency list altogether. This solution should ensure that we never install C++ wheels except in environments where they are needed.
As a consequence of this, we will always have C++ wheels that contain DSOs (no more empty shells) and they can always default to loading the library from the wheel. They can also be consistently configured to allow loading system libraries (e.g. using an environment variable).
The text was updated successfully, but these errors were encountered:
@trxcllnt pointed out that we could just reuse the use_cuda_wheels flag since that already pretty much covers this (we may want to rename it to capture the more general nature though).
Currently C++ wheels are part of the devcontainer manifest and are therefore among the libraries built in those environments. However, in these environments the wheel build is designed to be a no-op because the library is already installed. The implementation involves 1) the CMake for the C++ exiting early if a
find_package
call finds that the library has already been built elsewhere, 2) the Python logic forload_library
allowing through various cases where no library is found under the assumption that the Python extension modules will have suitable RPATHs baked in to find libraries, and 3) various dependency shenanigans to ensure that we get the desired dependency lists in different cases (e.g. DLFW vs. pip devcontainers vs. conda devcontainers). These changes require us to jump through a number of hoops downstream because they force us to consider cases where a C++ wheel package may be installed but actually not contain any DSO. With the benefit of some experience, it's time that we reimagine how we handle these various cases.Rather than doing what we currently do, we can also satisfy all of these requirements by simply not installing the C++ wheels at all into environments where the system library is always preferred. The difficulty with accomplishing this so far in devcontainers and DLFW has been that this conflicts with the declared dependency lists, which state that e.g. pylibcudf depends on libcudf. Trying to remove the libcudf installation results in pip thinking the dependency is unsatisfied and going to install it. We can fix this by changing our dependencies.yaml files to ensure that when building with certain settings (config-settings for RBB that are passed to dfg's dependency generator) the C++ wheels are filtered out of the dependency list altogether. This solution should ensure that we never install C++ wheels except in environments where they are needed.
As a consequence of this, we will always have C++ wheels that contain DSOs (no more empty shells) and they can always default to loading the library from the wheel. They can also be consistently configured to allow loading system libraries (e.g. using an environment variable).
The text was updated successfully, but these errors were encountered: