Rapids Shuffle Manager on Anaconda3, Missing UCX library

[leehaoun]

I submitted the job with Shuffle Manager turned on. However, the code isn't executing as it fails to find the 'org.openucx.jucx.ucp.UcpConstants' related to UCX library. UCX is definitely installed in the Anaconda3 environment, and the output of 'ucx_info -v' is as follows and ucx_perftest work well.

# Library version: 1.14.1
# Library path: /root/anaconda3/envs/rapids-23.10/bin/../lib/libucs.so.0
# API headers version: 1.14.1
# Git branch '', revision 16941ea
# Configured with: --disable-logging --disable-debug --disable-assertions --disable-
params-check` --build=x86_64-conda-linux-gnu --host=x86_64-conda-linux-gnu --prefix=/root/anaconda3/envs/rapids-23.10 --with-sysroot --enable-cma --enable-mt --enable-numa --with-gnu-ld --with-cuda=/usr/local/cuda --with-rdmacm=/root/anaconda3/envs/rapids-23.10 --with-verbs=/root/anaconda3/envs/rapids-23.10

below is error log

py4j.protocol.Py4JJavaError: An error occurred while calling o146.showString.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 7 in stage 0.0 failed 4 times, most recent failure: Lost task 7.3 in stage 0.0 (TID 31) (30.0.0.1 executor 0): java.lang.NoClassDefFoundError: Could not initialize class org.openucx.jucx.ucp.UcpConstants
        at org.openucx.jucx.ucp.UcpParams.requestTagFeature(UcpParams.java:100)
        at com.nvidia.spark.rapids.shuffle.ucx.UCX.<init>(UCX.scala:75)
        at com.nvidia.spark.rapids.shuffle.ucx.UCXShuffleTransport.ucx$lzycompute(UCXShuffleTransport.scala:72)
        at com.nvidia.spark.rapids.shuffle.ucx.UCXShuffleTransport.ucx(UCXShuffleTransport.scala:70)

I think Rapids is having trouble finding the library path of the installed UCX. (The code runs smoothly without any issues when excluding operations like 'Join' that perform shuffling.) How can I make Rapids recognize the library path of UCX?

[abellina]

@leehaoun agree. I think you want to set spark.executorEnv.LD_LIBRARY_PATH=/root/anaconda3/envs/rapids-23.10/lib in the executors for this case. The reason is that anaconda overrides env variables to make an environment active by sourcing a script. There may be a cleaner way but this should work.