NekRS on Cluster

[philipJFpfahl]

Hello,

Recently we installed Cardinal with NekRS, openMC, and DAGMC on our cluster.
I ran into some trouble while doing the tutorial.
Every tutorial step seems to run (also the other coupled tutorials), except for the MSFR. All the files are added to the folder (I hope). If I run the openMC.i file it runs a bit and crashes when starting nekRS so I only run the nek.i file.
So I get the error below.

$ mpiexec -n 100 ../../cardinal_new/cardinal/cardinal-opt -i nek.i

  ____   ___   / /__ / __ \/ ___/
  / __ \ / _ \ / //_// /_/ /\__ \ 
 / / / //  __// ,<  / _, _/___/ / 
/_/ /_/ \___//_/|_|/_/ |_|/____/  v23.0.7 (907edeac)

COPYRIGHT (c) 2019-2023 UCHICAGO ARGONNE, LLC

MPI tasks: 100

Initializing device 
reading par file ...
general::filtering is deprecated and might be removed in the future!
general::filtermodes is deprecated and might be removed in the future!
general::filterweight is deprecated and might be removed in the future!
active occa mode: Serial

using NEKRS_HOME: /nfs/safe_shared/cardinal/install
using NEKRS_CACHE_DIR: /nfs/safe_shared/philip_share/msfr/.cache
using OCCA_CACHE_DIR: /nfs/safe_shared/philip_share/msfr/.cache/occa/

building nekInterface for lx1=8, lelt=6163 and lelg=616000 ... 
/nfs/safe_shared/cardinal/install/nek5000/core/map2.f:184:29:

  184 |       call read_con(wk4,size(wk),neli,nlv,ierr)
      |                             1
Error: Result of SIZE overflows its kind at (1)
/nfs/safe_shared/cardinal/install/nek5000/core/map2.f:189:30:

  189 |         call find_con(wk,size(wk),tol,ierr)
      |                              1
Error: Result of SIZE overflows its kind at (1)
/nfs/safe_shared/cardinal/install/nek5000/core/map2.f:192:32:

  192 |           call find_con(wk,size(wk),tol,ierr)
      |                                1
Error: Result of SIZE overflows its kind at (1)
make: *** [/nfs/safe_shared/philip_share/msfr/.cache/nek5000/makefile:167: /nfs/safe_shared/philip_share/msfr/.cache/nek5000/obj/map2.o] Error 1
make: *** Waiting for unfinished jobs....
Error in buildNekInterface: 
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 18 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------

I hope you can help with this issue.

Thanks in advance and have a nice weekend

Philip

[aprilnovak]

Hi @philipJFpfahl I am sorry I missed your question and took so long to respond. For some reason I do not think Github sent me an email about it.

The MSFR case is our largest tutorial - those types of error messages mean you are running out of memory. Can you tell me a bit more about what HPC system you are running on?

[philipJFpfahl]

Hello,
No worries, it is also not a super high priority.

The cluster has 3 nodes with:

(base) [pfp01@node02 ~]$ lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              256
On-line CPU(s) list: 0-255
Thread(s) per core:  2
Core(s) per socket:  64
Socket(s):           2
NUMA node(s):        2
Vendor ID:           AuthenticAMD
CPU family:          25
Model:               1
Model name:          AMD EPYC 7763 64-Core Processor
Stepping:            1
CPU MHz:             2450.000
CPU max MHz:         3529,0520
CPU min MHz:         1500,0000
BogoMIPS:            4890.78
Virtualization:      AMD-V
L1d cache:           32K
L1i cache:           32K
L2 cache:            512K
L3 cache:            32768K
NUMA node0 CPU(s):   0-63,128-191
NUMA node1 CPU(s):   64-127,192-255

and

              total        used        free      shared  buff/cache   available
Mem:          1,0Ti       3,9Gi       977Gi        37Mi        25Gi       997Gi
Swap:         4,0Gi        63Mi       3,9Gi

So there is one Tb of RAM for each node with 256 cores.
I can run this slurm job:

1 #!/bin/bash
2 #SBATCH --job-name=nekRS            # create a short name for your job
3 #SBATCH -N 3                        # node count
4 #SBATCH -n 768                      # total number of tasks across all nodes
5 #SBATCH --mem=875g
6 
7 module purge
8 
9 module load cardinal
10 
11 mpiexec -n 768 ../../cardinal_new/cardinal/cardinal-opt -i nek.i  

and still get the error.

How much memory do you allocate when running the problem?

All the best,
Philip

[aprilnovak]

I ran this case on Summit, about 25 nodes. Each node has 2 Power9 processors, for ~500 GB RAM. The mesh for this case has 600,000 elements, which gives 1.2 billion grid points.

[aprilnovak]

You can try decreasing the polynomial order, and at some point you should be able to run on that machine, if you want to test things out.

[philipJFpfahl]

Hello,

I finally got to test it and indeed, with reduced polynomial order the simulation ran.

Thank you :)

[nancysemwal]

@philipJFpfahl @aprilnovak

Hey I'm also running into issues while running this same MSFR tutorial. Could you help me out?

Since I do not have my HPC installation yet (working on it), I'm currently only focusing on running the openmc model. This is what I get when I do mpirun -np 4 openmc (executable path -> from cardinal/build/openmc/bin/openmc) with the tutorial's settings.xml, geometry.xml & materials.xml files:

Reading geometry XML file...
 WARNING: No graveyard volume found in the DagMC model. This may result in lost
          particles and rapid simulation failure.
 WARNING: No graveyard volume found in the DagMC model. This may result in lost
          particles and rapid simulation failure.
 WARNING: No graveyard volume found in the DagMC model. This may result in lost
          particles and rapid simulation failure.
Loading file msr.h5m
Initializing the GeomQueryTool...
Using faceting tolerance: 0.001```
Building acceleration data structures...
 Reading Li7 from
 /mnt/a1681f2d-e1f0-4952-96ed-275378d460c5/cross_sections/endfb-vii.1-hdf5/neutro
 n/Li7.h5
 WARNING: No graveyard volume found in the DagMC model. This may result in lost
          particles and rapid simulation failure.
          
 trimmed
 
 
 Initializing source particles...
 WARNING: Could not find the cell containing particle 12508

 ====================>     K EIGENVALUE SIMULATION     <====================

  Bat./Gen.      k            Average k
  =========   ========   ====================
 WARNING: Could not find the cell containing particle 37514
 WARNING: Could not find the cell containing particle 37532
 WARNING: Could not find the cell containing particle 12590
 WARNING: Could not find the cell containing particle 25054
 WARNING: Could not find the cell containing particle 37576
 WARNING: Could not find the cell containing particle 130
 WARNING: Could not find the cell containing particle 25107
 WARNING: Could not find the cell containing particle 37661
 WARNING: Could not find the cell containing particle 25162
 WARNING: Could not find the cell containing particle 12713
 WARNING: Could not find the cell containing particle 208
 WARNING: Could not find the cell containing particle 37730
 WARNING: Could not find the cell containing particle 243
 WARNING: Could not find the cell containing particle 12783
 WARNING: Could not find the cell containing particle 282
 WARNING: Could not find the cell containing particle 37781
 WARNING: Could not find the cell containing particle 37805
 WARNING: Could not find the cell containing particle 25343
 WARNING: Could not find the cell containing particle 25349
 WARNING: Could not find the cell containing particle 25358
 WARNING: Could not find the cell containing particle 37926
 WARNING: Could not find the cell containing particle 37933
 WARNING: Could not find the cell containing particle 438
 WARNING: Could not find the cell containing particle 12980
 WARNING: Could not find the cell containing particle 13009
 WARNING: Could not find the cell containing particle 13030
 WARNING: Could not find the cell containing particle 25475
 WARNING: Could not find the cell containing particle 13053
 WARNING: Could not find the cell containing particle 530
 WARNING: Could not find the cell containing particle 13077
 WARNING: Could not find the cell containing particle 38043
 ERROR: Maximum number of lost particles has been reached.

And the simulation gets aborted by MPI since the max lost particles threshold has been reached.
ChatGPT tells me that the warning about the missing graveyard volume could be due to an improperly validated msr.h5m file. Is that the case?

How do I proceed from here?

[nuclearkevin]

Hi @nancysemwal,

The DAGMC geometry used for this tutorial (msr.h5m) doesn't contain a graveyard volume, which is akin to the geometry not setting a vacuum boundary condition in OpenMC. This isn't a problem when running the tutorial through Cardinal (by running openmc.i with cardinal-opt) as the MoabSkinner automatically builds the graveyard volume for you (using the exodus mesh mirror). This doesn't happen when you run the OpenMC input files with a standalone OpenMC binary, and so the simulation rapidly fails due to lost particles that should be terminated by a vacuum boundary.

If you want to run this tutorial without nekRS you'll need to modify openmc.i to remove the [MultiApps] and [Transfers] block, then swap the executioner from type = Transient to type = Steady. The other alternative would be to regenerate the DAGMC geometry from the exodus mesh (msr.e) using Coreform Cubit to add a graveyard volume, but I would recommend against this course of action as it's rather tedious (and requires a Cubit license).

Hope that helps,
Kevin

[nancysemwal]

Hello @nuclearkevin

Thanks you for your time.

I could run just the openmc model by incorporating the changes in the openmc.i file. Also thanks for suggesting the alternative path using Cubit, that might come handy in the future.

Now I'm working on running the full coupled simulation once my HPC installation its complete today.