Build ~ LLNL Introduction

Introduction

An overview of Livermore Computing Resources can be found on their website. Details of building ABLATE for Quartz and Lassen is provided on separate with general tips below.

• Quartz
• Lassen

Lustre File System

Lustre is a high performance parallel distributed file system, developed under the Accelerated Strategic Computing Initiative and funded by the Department of Energy. It is one of the most common parallel distributed file systems used by HPC clusters in the TOP500.

At LLNL there are two main Lustre file systems:

   cd /p/lustre1/USERNAME

Can be used to access the older Lustre file system and

   cd /p/lustre2/USERNAME

Can be used to access the new optimized Lustre file system.

While Lustre has many good properties, like any tool it is important to understand how to best use it. The following are a few quick tips;

1. Never build executable files or libraries on a Lustre file system.
2. Avoid accessing multiple small files on Lustre file system, i.e. keep your input files on /usr/workspace/USERNAME for Quartz.
3. Limit the number of processes that write to the file system at any given time.

For a more detailed list of best practices and further explanations visit https://www.nas.nasa.gov/hecc/support/kb/lustre-best-practices_226.html

Spindle

Spindle is program designed to improve the performance of dynamic linking of HPC applications. While small runs may not see much a benefit from using Spindle, it never degrades performance. For running on Quartz, any run using more 43,200 processing elements(1200 nodes with 36 PE per node), Spindle should be used.

To use Spindle place the following command, on the same line and before srun in your SLURM script;

   spindle --location=/var/tmp/USERNAME 

A comple line would look like;

   spindle --location=/var/tmp/USERNAME srun -n36 /usr/workspace/USERNAME/ablateOpt/ablate --input $INPUT_FILE

To read further on Spindle: https://computing.llnl.gov/projects/spindle#:~:text=Spindle%20is%20a%20tool%20for,file%20system%20into%20a%20bottleneck.