Building on HPC cluster of CNAF-INFN

[mtrocadomoreira]

Hello!

I would like to build HiPACE on this cluster. Before I start building the profile.hipace configuration, I just wanted to ask if there are any obvious hardware-dependent flags I should be aware of.

The available GPUs in this cluster are NVIDIA K20, K40, K1 and V100 (see full cluster hardware specs here).

Thanks!

[SeverinDiederichs]

Hi Mariana,

thank you for reaching out. I would strongly recommend using the V100s, they are the most modern on the list. Do you know if they have 16 GB or 32 GB? That's not clear from the website.

The optimize for V100s, please add

export AMREX_CUDA_ARCH=7.0 # use 8.0 for A100 or 7.0 for V100

to your profile.hipace. Otherwise, you could also just add the flag -DAMREX_CUDA_ARCH=7.0 during compilation with cmake.

Please let us know if everything works, we could add the cluster to the documentation.

[mtrocadomoreira]

Thanks for the speedy reply!

Wow, it actually compiled successfully with a very simple configuration file, almost at the first try 🥹 Thank you to everyone who contributed to making this so easy to compile!

Here's what I used for the profile.hipace.cnaf-infn file:

module load compilers/cmake-3.27.7
module load compilers/gcc-12.3_sl7
module load compilers/cuda-9.1
module load compilers/openmpi-4-1-5_gcc12.3

export AMREX_CUDA_ARCH=7.0

export CC=$(which gcc)
export CXX=$(which g++)
export FC=$(which gfortran)

Let me come back to this thread once I have run a job successfully, to confirm that everything is flowing smoothly.

Do you know if they have 16 GB or 32 GB?

No, I don't... Do you think this could pose a limitation for larger simulations?

[SeverinDiederichs]

Happy to hear that the compilation worked!

Is there a newer version of cuda available on the cluster? Using a new compiler (GCC 12.3) but a very old cuda version (9.1) will probably not work. It would be great if there was something like cuda 11.8.

If it is not available, I'd suggest to ask the admins of the cluster whether they can install that.

No, I don't... Do you think this could pose a limitation for larger simulations?

16 GB could be a bit low for challenging simulations for AWAKE, 32 GB would be fine. You can certainly do quite a lot of things with 16 GB already but for what I assume you'd like to do 32 would be better.

If a simulation runs successfully on the GPU, it will tell you about how much memory it used.