Skip to content

Latest commit

 

History

History
86 lines (60 loc) · 3.55 KB

Readme_VEGA.md

File metadata and controls

86 lines (60 loc) · 3.55 KB

Building GEMINI on the ERAU HPC system

GEMINI must be built from a session on the login node as it requires git. The build system will not work in an interactive or queue session.

Load the necessary modules (cmake, gcc8, and openmpi):

module load cmake/3.20.0
module load gcc/8.3.0
module load openmpi/gcc8/64/3.1.2
module load python/3.7.7

Clone the core GEMINI repository, if needed. Navigate to the location where you want the source code (in this example we will use ~/Projects) and then clone:

cd ~/Projects
git clone https://github.com/gemini3d/gemini3d.git

Navigate into the source code directory and run cmake to prep your system for the build step:

cd gemini3d
cmake -B build

This script will take a few minutes to complete; it will note multiple missing packages but will download source code for these to be compiled during the build step. This step can take up to 10 minutes depending on download times. If this completes successfully you can move on to the compile step:

cmake --build build -j

A full compile will take approximate 5-10 minutes depending on which packages need to be compiled; you will see lots of warnings but these can be safely ignored. Executables are placed in the build directory and can be run from there.

Running GEMINI on the ERAU HPC

For a simulation not needing more than 256 GB of memory, 36 cores, and less than 24 hours of runtime, one should use an interactive session. This can be obtained by logging into VEGA and then doing:

qsub -I -l walltime=24:00:00 -l nodes=1:ppn=36

For longer or more expensive simulations you will need to use the queueing system, which requires a script that runs the code to be placed in the build directory where executables reside. An example of such a script is shown below:

#!/bin/bash
# This is comment
#PBS -q longq
#PBS -l walltime=120:00:00
#PBS -l nodes=10:ppn=36
#PBS -N GEMINI_mooreOK
# Not necessary to set wall time, 
# longq: 168 hours, 10 nodes (with 36proc for each), maximum 1 longq in queue per user
# normalq: 24 hours, 10 nodes, maximum 4 normalq in queue per user

# In you want to send job status to e-mail:
#PBS -M [email protected]
#PBS -m aeb
cd $PBS_O_WORKDIR

#PBS -e /scratch/zettergm/GEMINI3D/pbs_errors.out
#PBS -o /scratch/zettergm/GEMINI3D/pbs_output.out

# Load modules during job submission!
module load gcc/8.3.0
module load openmpi/gcc8/64/3.1.2

# Run the program
mpirun -np $PBS_NP ./gemini.bin ../simulations/mooreOK3D_hemis_medres_corrected_control/ -manual_grid 36 10 -debug

This simulation runs an example "mooreOK3D_hemis_medres" using 360 cores split in a process grid of 36x10 in the x2 and x3 directions.

Data management on the ERAU HPC

Once simulations are completed they need to be moved into your home directory, which is for long-term storage and has a 10 TB quota. Output data can also be downloaded for postprocessing and plotting as needed; "rsync" is probably the best utility for this (comes standard on unix-likes macOS and Linux). E.g. if you navigate to the directory on your local computer where you want to store the data (~/simulations in this example) you can copy data from remote VEGA directory ~/vega_simulations/testdata to your computer via:

cd ~/simulations/
rsync -av --progress [email protected]:vega_simulations/testdata ./

A benefit of using rsync is that if the data transfer is interrupted (bad connection) you can reissue the same command again and it will pick up where it left off and not unnecessarily transfer files it has already completed.