Skip to content

Latest commit

 

History

History
38 lines (25 loc) · 1.81 KB

README.md

File metadata and controls

38 lines (25 loc) · 1.81 KB
Raw

Bandwidth-Optimal, Fully-Offloaded Collectives

The paper will be presented at the SC '24 (The International Conference for High Performance Computing, Networking, Storage, and Analysis).

Artifact structure

  • CPU-driven Multicast-based Allgather/Broadcast integrated within the UCC library:
    • ./ucc/src/components/tl/spin - collectives backend source-code.
    • ./osu-micro-benchmarks-7.3 - OSU MPI benchmarks supporting per-iteration timing for Allgather/Broadcast collectives
    • ./ompi/ and ./ucx- subrepos with project dependencies
  • DPA-offloaded collective receive datapath PoC:
    • ./coll-offloading/*
    • ./coll-offloading/README.md - detailed manual for building/running PoC.
  • Multicast-based Allgather traffic model:
    • ./sim/estimate_allgather_cost.py

Building and running UCC-based collectives backend

Prerequisites: InfiniBand-based cluster and user permissions to create multicast-groups through OpenSM.

  1. $ git submodule update --init --recursive
  2. $ bash ./build.sh
  3. $ source ./sourceme.sh
  4. $ cd ./osu-micro-benchmarks-7.3/ && ./configure CC=mpicc CXX=mpicxx && make -j

OpenMPI 5.0.3 compiled with the support of UCC multicast backend will be stored in ./ompi/build. Multicast-based backend can be tested by running:

mpirun -x LD_LIBRARY_PATH -x PATH --hostfile <machine_hostfile> -np <nprocs> --mca coll_ucc_enable 1 --mca coll_ucc_priority 100 ./osu-micro-benchmarks-7.3/c/mpi/collective/blocking/osu_allgather -m 4096:8388608

DPA offloading

Prerequisites: Two servers with BlueField-3 NIC and DOCA v2.2.0.

See ./coll-offloading/README.md for building and running instructions instructions.

Traffic reduction with Multicast

Run python3 ./sim/estimate_allgather_cost.py to get .csv output with traffic estimations for different Allgather algorithms.