diff --git a/Docs/sphinx_doc/building.rst b/Docs/sphinx_doc/building.rst index eb50a4ba2..1777e9419 100644 --- a/Docs/sphinx_doc/building.rst +++ b/Docs/sphinx_doc/building.rst @@ -334,6 +334,99 @@ Finally, you can prepare your SLURM job script, using the following as a guide: To submit your job script, do ``sbatch [your job script]`` and you can check its status by doing ``squeue -u [your username]``. +AMReX--Kokkos on `Perlmutter`_ (NERSC) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There is an `example in amrex-devtests`_ of how one can build `Kokkos`_ and `AMReX`_. This uses ``cmake`` to first compile +Kokkos and AMReX, and then the example is built. This section describes the build procedure on `Perlmutter`_, though this can be used as +a template for other machines as well. + +Load the following modules, and specify ``MPI_INCLUDE_PATH`` in ``~/.bash_profile``. The ``cmake`` version has to be ``3.24.3``. + +.. code-block:: bash + + module load cray-mpich + module load PrgEnv-gnu + module load cudatoolkit/12.2 + module load cmake/3.24.3 + + export MPI_INCLUDE_PATH=/opt/cray/pe/mpich/8.1.28/ofi/gnu/12.3/include + +Make sure to do ``source ~/.bash_profile``. + +**Kokkos installation on Perlmutter** + +To install `Kokkos`_, execute the following commands. In the ``cmake`` command, specify the path where the Kokkos installation should reside +``-DCMAKE_INSTALL_PREFIX=``. The full path to the ``kokkos`` directory has to be specified in +``-DCMAKE_CXX_COMPILER=/bin/nvcc_wrapper``. + +.. code-block:: bash + + git clone https://github.com/kokkos/kokkos.git + cd kokkos + mkdir build + cd build + cmake .. -DCMAKE_INSTALL_PREFIX= -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_STANDARD=17 -DKokkos_ENABLE_CUDA=ON -DCMAKE_CXX_COMPILER=/bin/nvcc_wrapper -DKokkos_ENABLE_CUDA_RELOCATABLE_DEVICE_CODE=ON -DKokkos_ARCH_PASCAL60=ON + make -j8 + make install + +**AMReX installation on Perlmutter** + +.. note:: + + After cloning the repository in the first step below, add the following lines to ``amrex/CMakeLists.txt`` for MPI installation. + + .. code-block:: bash + + # Find MPI + find_package(MPI REQUIRED) + + # Include MPI headers + include_directories(${MPI_INCLUDE_PATH}) + +To install `AMReX`_, execute the following commands. In the ``cmake`` command, specify the path where the installation AMReX installation should reside +``-DCMAKE_INSTALL_PREFIX=``. + +.. code-block:: bash + + git clone https://github.com/AMReX-Codes/amrex.git + cd amrex + mkdir build + cd build + cmake .. -DCMAKE_INSTALL_PREFIX= -DAMReX_GPU_BACKEND=CUDA -DAMReX_CUDA_ARCH=60 -DAMReX_MPI=OFF -DCMAKE_PREFIX_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/23.9/math_libs/12.2/lib64 -DAMReX_MPI=ON -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpicxx -DMPI_INCLUDE_PATH=/opt/cray/pe/mpich/8.1.28/ofi/gnu/12.3/include + make -j8 + make install + +**Compiling the AMReX-Kokkos example on Perlmutter** + +.. note:: + + After cloning the repository in the first step below, add the following lines to ``amrex-devtests/kokkos/CMakeLists.txt`` for MPI installation. + + .. code-block:: bash + + # Find MPI + find_package(MPI REQUIRED) + + # Include MPI headers + include_directories(${MPI_INCLUDE_PATH}) + +To compile the AMReX-Kokkos example, execute the following commands. In the ``cmake`` command, specify the full path to the amrex installation +directory ``-DAMReX_ROOT=``, and the Kokkos installation directory ``-DKokkos_ROOT=``. + +.. code-block:: bash + + git clone https://github.com/WeiqunZhang/amrex-devtests.git + cd amrex-devtests/kokkos + mkdir build + cd build + cmake .. -DENABLE_CUDA=ON -DAMReX_ROOT= -DKokkos_ROOT= -DCMAKE_CUDA_ARCHITECTURES=60 -DMPI_INCLUDE_PATH=/opt/cray/pe/mpich/8.1.28/ofi/gnu/12.3/include + make -j8 + +.. _`example in amrex-devtests`: https://github.com/WeiqunZhang/amrex-devtests/tree/main/kokkos +.. _`Kokkos`: https://github.com/kokkos/kokkos +.. _`AMReX`: https://github.com/AMReX-Codes/amrex +.. _`Perlmutter`: https://docs.nersc.gov/systems/perlmutter/architecture/ Kestrel (NREL) ~~~~~~~~~~~~~~