Evaluate replacing conda-build with rattler-build

[vyasr]

RAPIDS currently builds conda packages in CI using conda-build. The rattler-build tool is a newer alternative. It is written in Rust, and should be faster than conda-build (I haven't seen any official benchmarks yet, though). It only supports a limited subset of the meta.yaml recipe format, but that subset is designed to still enable all the same features, just with a more limited syntax (see CEPS 13 and 14). conda-build overhead is nontrivial (I've never benchmarked it, but I know it can stretch into multiple minutes beyond the environment solve when doing local CI reproductions), and reducing that would be quite valuable for us in improving our CI turnaround. Moreover, switching to the more restricted syntax described in the above CEPs would be beneficial because it would convert our conda recipes into pure YAML rather than the extended YAML currently used by meta.yaml. That change is important because the YAML extensions currently in our recipe make it impossible to parse or write with standard YAML parsers, which is a big reason why we have struggled to do things like support meta.yaml files in rapids-dependency-file-generator.

We should do a PoC of replacing conda-build with rattler-build in one repo (preferably something reasonably complex like cudf or cugraph) to see what it would take to make this transition, and how much we would benefit.

[vyasr]

I now have two POCs of using rattler-build

• Build conda packages using rattler-build ucxx#248
• Replace conda-build with rattler-build cugraph#4551

Some notes:

• rattler-build requires strict channel priority (Allow to change channel priority prefix-dev/rattler-build#343) so we need to address Switch to using strict channel priority during RAPIDS builds #84 first
• Our general model of "build-once, install into separate outputs" requires a feature that rattler-build still marks as experimental, the multi-output cache. This doesn't personally concern me all that much since they are developing quite fast and I expect bug fixes to happen far more quickly than with conda-build, but it is something to be aware of.
• There are some issues around cache run exports that would be best to have addressed upstream before we adopt.
  • Update: This is now fixed in the latest release (0.19).

I've littered the PRs liberally with comments, so those should provide additional information.

[msarahan]

Nice! Looks like you got some pretty good speedup there - maybe around 5 minutes of speedup, from what I can see. Considering the whole build time was 8-10 minutes, saving 5 minutes is huge.

[vyasr]

Yup, similar speedups in absolute numbers for cugraph (smaller percentage, but still not bad to go from 28 mins to 22 mins. Also there's probably an extra 30 s in there for installing rattler-build itself in each job since it's not yet preinstalled in the image (but it will be).

[bdice]

I looked at the two POCs for ucxx and cugraph and I am supportive of this effort. It seems like there are a lot of steps to take first (like fixing strict priority conda builds, maybe altering librmm run exports, and maybe some rattler-build issues -- basically all the things that are marked in the comments) but overall I think this is mature enough that we can invest some time in this direction.

[vyasr]

FYI @wolfv @ruben-arts here's the follow-up to our discussions from SciPy 🙂

[vyasr]

We're actively discussing #84, which is IMO a blocker to making this change. We can work around it, but it requires a lot of extra boilerplate in recipes that I don't think it's worth proliferating.

[msarahan]

Adding repo tracker for progress tracking:

• rmm
• raft
• cudf
• kvikio
• cuml
• cucim
• cugraph (Replace conda-build with rattler-build cugraph#4551)
• cugraph-ops
• cumlprims_mg
• cuspatial
• cuvs
• cuxfilter
• dask-cuda
• rapids-cmake
• rapids-dask-dependency
• ucx-py
• ucxx(Build conda packages using rattler-build ucxx#248)
• wholegraph