Adding docs for AMReX-Kokkos on Perlmutter (#1889)

Co-authored-by: Mahesh Natarajan <[email protected]>

Docs/sphinx_doc/building.rst

@@ -334,6 +334,99 @@ Finally, you can prepare your SLURM job script, using the following as a guide:
 
 To submit your job script, do ``sbatch [your job script]`` and you can check its status by doing ``squeue -u [your username]``.
 
+AMReX--Kokkos on `Perlmutter`_ (NERSC)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There is an `example in amrex-devtests`_ of how one can build `Kokkos`_ and `AMReX`_. This uses ``cmake`` to first compile
+Kokkos and AMReX, and then the example is built. This section describes the build procedure on `Perlmutter`_, though this can be used as
+a template for other machines as well.
+
+Load the following modules, and specify ``MPI_INCLUDE_PATH`` in ``~/.bash_profile``. The ``cmake`` version has to be ``3.24.3``.
+
+.. code-block:: bash
+
+   module load cray-mpich
+   module load PrgEnv-gnu
+   module load cudatoolkit/12.2
+   module load cmake/3.24.3
+
+   export MPI_INCLUDE_PATH=/opt/cray/pe/mpich/8.1.28/ofi/gnu/12.3/include
+
+Make sure to do ``source ~/.bash_profile``.
+
+**Kokkos installation on Perlmutter**
+
+To install `Kokkos`_, execute the following commands. In the ``cmake`` command, specify the path where the Kokkos installation should reside
+``-DCMAKE_INSTALL_PREFIX=<path-to-kokkos-install-dir>``. The full path to the ``kokkos`` directory has to be specified in
+``-DCMAKE_CXX_COMPILER=<full-path-to-kokkos-dir>/bin/nvcc_wrapper``.
+
+.. code-block:: bash
+
+   git clone https://github.com/kokkos/kokkos.git
+   cd kokkos
+   mkdir build
+   cd build
+   cmake .. -DCMAKE_INSTALL_PREFIX=<path-to-kokkos-install-dir> -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_STANDARD=17 -DKokkos_ENABLE_CUDA=ON -DCMAKE_CXX_COMPILER=<full-path-to-kokkos-dir>/bin/nvcc_wrapper -DKokkos_ENABLE_CUDA_RELOCATABLE_DEVICE_CODE=ON -DKokkos_ARCH_PASCAL60=ON
+   make -j8
+   make install
+
+**AMReX installation on Perlmutter**
+
+.. note::
+
+   After cloning the repository in the first step below, add the following lines to ``amrex/CMakeLists.txt`` for MPI installation.
+
+   .. code-block:: bash
+
+      # Find MPI
+      find_package(MPI REQUIRED)
+
+      # Include MPI headers
+      include_directories(${MPI_INCLUDE_PATH})
+
+To install `AMReX`_, execute the following commands. In the ``cmake`` command, specify the path where the installation AMReX installation should reside
+``-DCMAKE_INSTALL_PREFIX=<path-to-amrex-install-dir>``.
+
+.. code-block:: bash
+
+   git clone https://github.com/AMReX-Codes/amrex.git
+   cd amrex
+   mkdir build
+   cd build
+   cmake .. -DCMAKE_INSTALL_PREFIX=<path-to-amrex-install-dir> -DAMReX_GPU_BACKEND=CUDA -DAMReX_CUDA_ARCH=60 -DAMReX_MPI=OFF -DCMAKE_PREFIX_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/23.9/math_libs/12.2/lib64 -DAMReX_MPI=ON -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpicxx -DMPI_INCLUDE_PATH=/opt/cray/pe/mpich/8.1.28/ofi/gnu/12.3/include
+   make -j8
+   make install
+
+**Compiling the AMReX-Kokkos example on Perlmutter**
+
+.. note::
+
+   After cloning the repository in the first step below, add the following lines to ``amrex-devtests/kokkos/CMakeLists.txt`` for MPI installation.
+
+   .. code-block:: bash
+
+      # Find MPI
+      find_package(MPI REQUIRED)
+
+      # Include MPI headers
+      include_directories(${MPI_INCLUDE_PATH})
+
+To compile the AMReX-Kokkos example, execute the following commands. In the ``cmake`` command, specify the full path to the amrex installation
+directory ``-DAMReX_ROOT=<full-path-to-amrex-install-dir>``, and the Kokkos installation directory ``-DKokkos_ROOT=<full-path-to-kokkos-install-dir>``.
+
+.. code-block:: bash
+
+   git clone https://github.com/WeiqunZhang/amrex-devtests.git
+   cd amrex-devtests/kokkos
+   mkdir build
+   cd build
+   cmake .. -DENABLE_CUDA=ON -DAMReX_ROOT=<full-path-to-amrex-install-dir> -DKokkos_ROOT=<full-path-to-kokkos-install-dir> -DCMAKE_CUDA_ARCHITECTURES=60 -DMPI_INCLUDE_PATH=/opt/cray/pe/mpich/8.1.28/ofi/gnu/12.3/include
+   make -j8
+
+.. _`example in amrex-devtests`: https://github.com/WeiqunZhang/amrex-devtests/tree/main/kokkos
+.. _`Kokkos`: https://github.com/kokkos/kokkos
+.. _`AMReX`: https://github.com/AMReX-Codes/amrex
+.. _`Perlmutter`: https://docs.nersc.gov/systems/perlmutter/architecture/
 
 Kestrel (NREL)
 ~~~~~~~~~~~~~~