One problem about MPI and CUDA

[lijiangdong1999]

Hello, I use spack to install warpx with mpi and cuda in my cluster. I want to run it with some gpus, but I find it can only run many ntasks with one gpu.
Followed is my problem.
You can see if ntask is such as 4, then one gpu is used with 4 ntasks.


If ntask is 1, then one gpu is used with 1 ntask.


It's very strange.
What I want is that one node run many gpus, and one gpu run one ntask. Do you have some solution? Thanks in advance.

[ax3l]

Hi @lijiangdong1999 , this looks like a slurm pinning issue, as if your MPI processes are all assigned the same GPU.

It is best to coordinate with your cluster admins to get a 1-MPI-rank to 1-exclusive-GPU pinning. Try for instance this with Slurm:

# ...
#SBATCH --exclusive
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-node=8
# ...

instead of your --gres=gpu:N flag. Does that help?

Otherwise, can you share more details, e.g., the documentation page of the cluster you are on?

[lijiangdong1999]

I am sorry, it doesn't help. The cluster I use is built by my colleague. I can only give you some information about my cluster.

[lijiangdong1999]

I am sorry, it doesn't help. The cluster I use is built by my colleague. I can only give you some information about my cluster.
server_info.txt

[lijiangdong1999]

@ax3l This is the information about my cluster. Thanks in advance.