add libcuvs Python builds #440
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jameslamb
added
breaking
jameslamb
commented
Jan 21, 2025
features/src/rapids-build-utils/opt/rapids-build-utils/manifest.yaml
Outdated
Show resolved
Hide resolved
|
🎉 builds succeeded here! (build link) That's strong evidence that rapidsai/cuvs#594 won't break devcontainers. |
bdice
approved these changes
Jan 22, 2025
Co-authored-by: James Lamb <[email protected]>
|
Discussed with @jameslamb -- we will go ahead and merge this to unblock rapidsai/cuvs#594. |
rapids-bot bot
pushed a commit
to rapidsai/cuvs
that referenced
this pull request
Jan 23, 2025
Contributes to rapidsai/build-planning#33. Proposes packaging `libcuvs` as a wheel, which is then re-used by `cuvs-cu{11,12}`. Similar changes were recently made in RAFT: rapidsai/raft#2531 As part of this, also proposes: * introducing a new CMake option, `CUVS_COMPILE_DYNAMIC_ONLY`, to allow building/installing only the dynamic shared library (i.e. skipping the static library) * enforcing `rapids-cmake`'s preferred CMake style (similar rapidsai/raft#2531 (comment)) * standardizing `clang` pins across the project, and pinning to `clang` 16 for Rust builds (#594 (comment)) ## Notes for Reviewers ### Benefits of these changes * smaller wheels for projects that depend on cuVS (they can dynamically link against `libcuvs` instead of statically linking in the pieces they need) * fewer CI resources used for cuVS wheels (no more re-compiling for every Python minor version) * faster, cheaper cuML wheel builds (rapidsai/cuml#6199 (comment)) * other benefits mentioned in rapidsai/build-planning#33 ### Wheel contents `libcuvs`: * `libcuvs.so` and `libcuvs_c.so` (shared library) * cuVS headers * vendored dependencies (hnswlib) `cuvs`: * `cuvs` Python / Cython code and compiled Cython extensions ### Size changes (CUDA 12, Python 3.12, x86_64) | wheel | num files (before) | num files (this PR) | size (before) | size (this PR) | |:---------------:|------------------:|-----------------:|--------------:|---------------:| | `libcuvs` | --- | 67 | --- | 843M | | `cuvs` | 88 | 84 |845M | 2M | |**TOTAL** | **88** | **131** | **845M** | **845M** | *NOTES: size = compressed, "before" = 2025-01-22 nightlies* <details><summary>how I calculated those (click me)</summary> * nightly commit = rapidsai/cuml@7c715c4 * PR = this PR ```shell docker run \ --rm \ --network host \ --env RAPIDS_NIGHTLY_DATE=2025-01-22 \ --env CUVS_NIGHTLY_SHA=f1de1b2 \ --env CUVS_PR="pull-request/594" \ --env CUVS_PR_SHA="97c56178cd0e07e4b6b138bb0904af78379f1bb3" \ --env RAPIDS_PY_CUDA_SUFFIX=cu12 \ --env WHEEL_DIR_BEFORE=/tmp/wheels-before \ --env WHEEL_DIR_AFTER=/tmp/wheels-after \ -it rapidsai/ci-wheel:cuda12.5.1-rockylinux8-py3.12 \ bash # --- nightly wheels --- # mkdir -p ./wheels-before export RAPIDS_BUILD_TYPE=branch export RAPIDS_REF_NAME="branch-25.02" # cuvs RAPIDS_PY_WHEEL_NAME="cuvs_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/cuvs \ RAPIDS_SHA=${CUVS_NIGHTLY_SHA} \ rapids-download-wheels-from-s3 python ./wheels-before # --- wheels from CI --- # mkdir -p ./wheels-after export RAPIDS_BUILD_TYPE="pull-request" # libcuvs RAPIDS_PY_WHEEL_NAME="libcuvs_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/cuvs \ RAPIDS_REF_NAME="${CUVS_PR}" \ RAPIDS_SHA="${CUVS_PR_SHA}" \ rapids-download-wheels-from-s3 cpp ./wheels-after # cuvs RAPIDS_PY_WHEEL_NAME="cuvs_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/cuvs \ RAPIDS_REF_NAME="${CUVS_PR}" \ RAPIDS_SHA="${CUVS_PR_SHA}" \ rapids-download-wheels-from-s3 python ./wheels-after pip install pydistcheck pydistcheck \ --inspect \ --select 'distro-too-large-compressed' \ ./wheels-before/*.whl \ | grep -E '^checking|files: | compressed' \ > ./before.txt # get more exact sizes du -sh ./wheels-before/* pydistcheck \ --inspect \ --select 'distro-too-large-compressed' \ ./wheels-after/*.whl \ | grep -E '^checking|files: | compressed' \ > ./after.txt # get more exact sizes du -sh ./wheels-after/* ``` </details> ### How I tested this * rapidsai/devcontainers#440 * rapidsai/cuml#6199 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) - Ben Frederickson (https://github.com/benfred) URL: #594
vyasr
pushed a commit
that referenced
this pull request
Jan 23, 2025
Contributes to rapidsai/build-planning#33 Adjusts `rapids-build-utils` manifest for release 25.02 to account for the introduction of new `libcuml` wheels (rapidsai/cuml#6199). ## Notes for Reviewers This shouldn't be merged still pointing at my forks. Plan: 1. merge #440, then update this branch 2. see CI pass here 3. see all CI except devcontainers pass on rapidsai/cuml#6199 4. point this PR back at upstream and admin-merge it 5. re-run devcontainers CI on rapidsai/cuml#6199 and see it pass
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Contributes to rapidsai/build-planning#33
Adjusts
rapids-build-utilsmanifest for release 25.02 to account for the introduction of newlibcuvswheels (rapidsai/cuvs#594).Notes for Reviewers
This shouldn't be merged still pointing at my forks. Plan: