WIP: [DO NOT MERGE] introduce libraft wheels

[jameslamb]

Replaces #2360, contributes to rapidsai/build-planning#33.

Proposes packaging libraft as a wheel, which is then re-used by:

• pylibraft-cu{11,12} and raft-cu{11,12} (this PR)
• libcugraph-cu{11,12} in WIP: [DO NOT MERGE] introduce libcugraph wheels cugraph#4804
• libcuml-cu{11,12} in WIP: [DO NOT MERGE] introduce libcuml wheels cuml#6199

Notes for Reviewers

If you see this note, that means this is not ready for review.

Dependency Flows

libraft wheels contain all of:

• raft headers (and everything they require)
• libraft.a (static library)
• libraft.so (shared object)

libcugraph and libcuml need libraft at build time to statically link against libraft.a.

pylibcugraph, cugraph, and cuml need libraft at build time just to get the RAFT headers (since those are used in libcugraph's and libcuml's public headers).

---
title: Build dependencies
---
flowchart TD
    A[libraft] -->B[pylibraft]
    A --> C[raft-dask]
    A --> D[libcugraph]
    D --> E[pylibcugraph]
    D --> F[cugraph]
    A --> G[libcuml]
    A --> H[cuml]
    A --> E
    B --> E
    A --> F
    B --> F
    E --> F
    G --> H

Loading

Nothing in cugraph would need libraft at runtime, because libcugraph is statically linked against libraft.

---
title: Runtime dependencies
---
flowchart LR
    A[libraft] --> B[pylibraft]
    A --> C[raft-dask]

Loading

Presumably wholegraph could follow a similar pattern

Size changes (CUDA 12, Python 3.12, x86_64)

wheel	num files (before)	num files (after)	size (before)	size (this PR)
libraft.	---	---	---	---
pylibraft	---	---	---	---
raft-dask	---	---	---	---
libcugraph	---	---	---	---
pylibcugraph	190	---	900M	---
cugraph	314	---	901M	---
libcuml	---	---	---	---
cuml	---	---	---	---
TOTAL	****	---	****	---

NOTES: size = compressed, "before" = 2024-12-17 nightlies (rapidsai/cugraph@5c8f850), cugraph libraries from rapidsai/cugraph#4804

how I calculated those (click me)

docker run \
    --rm \
    -v $(pwd):/opt/work:ro \
    -w /opt/work \
    --network host \
    --env RAPIDS_NIGHTLY_DATE=2024-12-06 \
    --env RAPIDS_NIGHTLY_SHA=5c8f850 \
    --env RAPIDS_PR_NUMBER=4804 \
    --env RAPIDS_PY_CUDA_SUFFIX=cu12 \
    --env RAPIDS_REPOSITORY=rapidsai/cugraph \
    --env WHEEL_DIR_BEFORE=/tmp/wheels-before \
    --env WHEEL_DIR_AFTER=/tmp/wheels-after \
    -it rapidsai/ci-wheel:cuda12.5.1-rockylinux8-py3.12 \
    bash

mkdir -p "${WHEEL_DIR_BEFORE}"
mkdir -p "${WHEEL_DIR_AFTER}"

cpp_projects=(
    libcugraph
)
py_projects=(
    cugraph
    pylibcugraph
)

for project in "${py_projects[@]}"; do
    # before
    RAPIDS_BUILD_TYPE=nightly \
    RAPIDS_PY_WHEEL_NAME="${project}_${RAPIDS_PY_CUDA_SUFFIX}" \
    RAPIDS_REF_NAME="branch-25.02" \
    RAPIDS_SHA=${RAPIDS_NIGHTLY_SHA} \
        rapids-download-wheels-from-s3 python "${WHEEL_DIR_BEFORE}"

    # after
    #RAPIDS_BUILD_TYPE=pull-request \
    #RAPIDS_PY_WHEEL_NAME="${project}_${RAPIDS_PY_CUDA_SUFFIX}" \
    #RAPIDS_REF_NAME="pull-request/${RAPIDS_PR_NUMBER}" \
    #    rapids-download-wheels-from-s3 python "${WHEEL_DIR_AFTER}"
done

for project in "${cpp_projects[@]}"; do    
    # after
    RAPIDS_BUILD_TYPE=pull-request \
    RAPIDS_PY_WHEEL_NAME="${project}_${RAPIDS_PY_CUDA_SUFFIX}" \
    RAPIDS_REF_NAME="pull-request/${RAPIDS_PR_NUMBER}" \
        rapids-download-wheels-from-s3 cpp "${WHEEL_DIR_AFTER}"
done

pip install pydistcheck
pydistcheck \
    --inspect \
    --select 'distro-too-large-compressed' \
    ${WHEEL_DIR_BEFORE}/*.whl

pydistcheck \
    --inspect \
    --select 'distro-too-large-compressed' \
    ${WHEEL_DIR_AFTER}/*.whl

[jameslamb]

/ok to test

[jameslamb]

/ok to test

[jameslamb]

/ok to test

[jameslamb]

/ok to test

[jameslamb]

/ok to test

[jameslamb]

/ok to test