Are the dependency constraints on Pytorch still applicable?

[bastonero]

Dear Developers,
Thanks first of all for the amazing work, it is really useful and easy to use.

I am trying to install Nequip on LUMI-G, where they suggest to install Python packages via Singularity containers. Unfortunately, the ROCM version of those is already greater than what is compatible for Pytorch 1.10.0 < version < 1.12.0. It would be great to know if these constaints are still applicable, or you already did some tests (e.g. using Pytorch 2.x).

Thanks again for any help.

[Linux-cpp-lisp]

Hi @bastonero ,

Thank you, glad to hear!

I think it has mostly been established that newer PyTorch works fine for AMD GPUs, for which some relevant threads are:

• Training slowdown #311
• Allegro LAMMPS on AMD GPUs pair_allegro#23
• ERROR: CUDA_tensor_histogram failed to get free global memory when using `nequip-train` #402

The newest versions may also be fine on CUDA, but that has not been completely confirmed yet. 1.* versions that are not 1.11 should be avoided. If you encounter this first issue, or find anything that works (or doesn't work) unexpectedly, please post that info here!

[henriasv]

Hi, on LUMI-G, there is already a container available for nequip and pair_allegro. The procedure is supposed to be the following:

wget https://462000008.lumidata.eu/easyconfigs/LAMMPS-stable-12Aug2023-update2-pair-allegro-rocm-5.2.3-pytorch-1.13-20240303.eb
Then, for example (or wherever it is convenient to store builds:
export EBU_USER_PREFIX=/project/{your_project}/EasyBuild

Then:

module load LUMI/23.09
module load partition/container
module load EasyBuild-user

eb LAMMPS-stable-12Aug2023-update2-pair-allegro-rocm-5.2.3-pytorch-1.13-20240303.eb

Now you can do (typically before nequip-train or lmp -in ... in your job script):

module purge
module load LUMI/23.09
module load LAMMPS/stable-12Aug2023-update2-pair-allegro-rocm-5.2.3-pytorch-1.13-20240303

With these modules loaded, commands nequip-train, equip-evaluate, nequip-deploy, nequip-benchmark and lmp (lammps with pair allegro) are available.

Typical job script using lammps:

#!/bin/bash
#SBATCH --job-name=lammps
#SBATCH --time=0-04:00:00
#SBATCH --mem-per-cpu=8000
#SBATCH --cpus-per-task=4
#SBATCH --ntasks=1
#SBATCH --account={your_project}
#SBATCH --gpus=1
#SBATCH --partition=small-g

# Load modules
module load LUMI/23.09
module load LAMMPS/stable-12Aug2023-update2-pair-allegro-rocm-5.2.3-pytorch-1.13-20240303

# Setup cpu-binding
# This part is only relevant when running with multiple GPUS. Make sure to ask for nodes, not just many gpus
CPU_BIND="map_cpu:49,57,17,25,1,9,33,41"

# Run program
srun lmp -in myrun.inp > stdout.out 2> stderr.out

Please let me know whether this works, and whether you are experiencing any problems. Need to write up this procedure for my team as well, so bonus if you can check whether it works.

[henriasv]

@bastonero For me the parallellization on a single node works (without actually using the cpu-binding I realised. Perhaps room for performance improvements) Did you try something similar to this? :

#!/bin/bash
#SBATCH --job-name=lammps
#SBATCH --time=0-01:00:00
#SBATCH --mem-per-cpu=8000
#SBATCH --cpus-per-task=4
#SBATCH --ntasks=8
#SBATCH --account=...
#SBATCH --nodes=1 --gpus-per-node=8
#SBATCH --partition=dev-g

set -o errexit

# Load modules
module load LUMI/23.09
module load LAMMPS/stable-12Aug2023-update2-pair-allegro-rocm-5.2.3-pytorch-1.13-20240303
set -x

# Run program
srun lmp -in 33e191f29ee1afb4c7d358fb8b4c030d4b1da1d6.inp > stdout.out 2> stderr.out

[bastonero]

@henriasv I tried it, also with kokkos flags. Apparently somehow there is an issue with Kokkos unable to see more than 1 GPU card per node. I contacted the LUMI support, I'll keep you posted if I find a solution working intra-node.

[bastonero]

@henriasv The support replied quickly, and the issue is apparently the fact that their select_gpu script would mask GPUs, and Allegro can see only one (for some reason). Either way, this makes the script even simpler. The following works for intra-node GPUs:

#!/bin/bash
#SBATCH --no-requeue
#SBATCH --job-name="test"
#SBATCH --get-user-env
#SBATCH --output=_scheduler-stdout.txt
#SBATCH --error=_scheduler-stderr.txt
#SBATCH --partition=dev-g
#SBATCH --account=project_{number}
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-node=8
#SBATCH --time=0-01:00:00
#SBATCH --exclusive
#SBATCH --hint=nomultithread

module purge
module load LUMI/23.09
module load LAMMPS/stable-12Aug2023-update2-pair-allegro-rocm-5.2.3-pytorch-1.13-20240303

CPU_BIND="map_cpu:49,57,17,25,1,9,33,41"
export MPICH_GPU_SUPPORT_ENABLED=1

srun --cpu-bind=${CPU_BIND} lmp -k on g 8 -sf kk -pk kokkos gpu/aware off neigh full -in in.lammps

[ozakary]

Hello @bastonero and @henriasv, apologies for the late question on this thread. I attempted to run LAMMPS with pair_allegro on LUMI-G following the procedure provided above by @henriasv (the container available for nequip and pair_allegro) and got the following error in the stderr.out file:

lmp: error while loading shared libraries: libamdhip64.so.6: cannot open shared object file: No such file or directory srun: error: nid005002: task 0: Exited with exit code 127 srun: Terminating StepId=8585979.0

The command lines used are:

export EBU_USER_PREFIX=/projappl/project_<project_ID>/EasyBuild

module load LUMI/23.09
module load partition/container
module load EasyBuild-user

eb LAMMPS-stable-12Aug2023-update2-pair-allegro-rocm-5.2.3-pytorch-1.13-20240303.eb

module purge
module load LUMI/23.09
module load LAMMPS/stable-12Aug2023-update2-pair-allegro-rocm-5.2.3-pytorch-1.13-20240303

I hope I didn't miss any command.

Your help in this issue will be greatly appreciated.

[henriasv]

Hi, the container installation has changed after the maintenance stop in September. I'll get back to you with updated instructions