Installing MOOSE on HPC - failed make in tests

[rnpratoori]

Check these boxes if you have followed the posting rules.

• I have consulted the Posting Guidelines.
• I have searched the Discussions Forum and MOOSE Framework Troubleshooting and have not found what I was looking for
• Q&A Getting Started is the most appropriate category for my question (trouble installing, beginner user, ...)

Issue or question about MOOSE

Hello,

I am trying to install MOOSE on HPC at Iowa State University (Nova). I was able to successfully install all the libraries. I did face an issue with libtirpc, but following this discussion, I disabled XDR requirement and everything was built successfully.

But, on running make -j 6 in tests, I got the following errors -
tests_make.log

It does look like it has something to do with libmesh, but could it be because of XDR? How do I resolve this issue?

(Optional) code in question / simulation log / errors

No response

Encountering Errors? Please include diagnostic output

No response

[milljm]

I think you may need to start over (rebuild both libMesh and MOOSE)? To me, this looks like an older libMesh (or perhaps a new one with an older MOOSE).

We had a very recent libMesh update roll through just yesterday. I feel this might be the issue.

[rnpratoori]

@GiudGiud This fixed it.

But I am facing the following errors during ./run_tests where it says mpiexec is not detected. There are also few failing because XDR was not detected but I guess this is expected because I removed the option.

restart/restartable_types.parallel/first: Working Directory: /work/wxia/share/software/MOOSE/projects/moose/test/tests/restart/restartable_types
restart/restartable_types.parallel/first: Running command: mpiexec -n 2 /work/wxia/share/software/MOOSE/projects/moose/test/moose_test-opt -i restartable_types.i --error --error-override --libtorch-device cpu
restart/restartable_types.parallel/first:
restart/restartable_types.parallel/first: ################################################################################
restart/restartable_types.parallel/first: Begin job output
restart/restartable_types.parallel/first:
restart/restartable_types.parallel/first: ################################################################################
restart/restartable_types.parallel/first: Python exception encountered
restart/restartable_types.parallel/first: ################################################################################
restart/restartable_types.parallel/first: Traceback (most recent call last):
restart/restartable_types.parallel/first:   File "/work/wxia/share/software/MOOSE/projects/moose/python/TestHarness/runners/SubprocessRunner.py", line 69, in spawn
restart/restartable_types.parallel/first:     self.process = subprocess.Popen(*process_args, **process_kwargs)
restart/restartable_types.parallel/first:   File "/opt/rit/el9/20230413/app/linux-rhel9-x86_64_v3/gcc-11.2.1/python-3.10.10-zwlkg4ljdinozolpn7os3bpbjvu7sw6m/lib/python3.10/subprocess.py", line 971, in __init__
restart/restartable_types.parallel/first:     self._execute_child(args, executable, preexec_fn, close_fds,
restart/restartable_types.parallel/first:   File "/opt/rit/el9/20230413/app/linux-rhel9-x86_64_v3/gcc-11.2.1/python-3.10.10-zwlkg4ljdinozolpn7os3bpbjvu7sw6m/lib/python3.10/subprocess.py", line 1847, in _execute_child
restart/restartable_types.parallel/first:     raise child_exception_type(errno_num, err_msg, err_filename)
restart/restartable_types.parallel/first: FileNotFoundError: [Errno 2] No such file or directory: 'mpiexec'
restart/restartable_types.parallel/first:
restart/restartable_types.parallel/first: The above exception was the direct cause of the following exception:
restart/restartable_types.parallel/first:
restart/restartable_types.parallel/first: Traceback (most recent call last):
restart/restartable_types.parallel/first:   File "/work/wxia/share/software/MOOSE/projects/moose/python/TestHarness/schedulers/Job.py", line 398, in try_catch
restart/restartable_types.parallel/first:     do()
restart/restartable_types.parallel/first:   File "/work/wxia/share/software/MOOSE/projects/moose/python/TestHarness/schedulers/Job.py", line 439, in <lambda>
restart/restartable_types.parallel/first:     spawn = lambda: self._runner.spawn(self.timer)
restart/restartable_types.parallel/first:   File "/work/wxia/share/software/MOOSE/projects/moose/python/TestHarness/runners/SubprocessRunner.py", line 71, in spawn
restart/restartable_types.parallel/first:     raise Exception('Error in launching a new task') from e
restart/restartable_types.parallel/first: Exception: Error in launching a new task
restart/restartable_types.parallel/first:
restart/restartable_types.parallel/first: ################################################################################
restart/restartable_types.parallel/first: End job output
restart/restartable_types.parallel/first:
restart/restartable_types.parallel/first ............................................ [min_cpus=2] FAILED (RUNNER SPAWN EXCEPTION)

[GiudGiud]

where do you see "mpiexec" not detected?
are you using the HPC test scehduler or just running tests on a single node?

[rnpratoori]

Actually, I noticed that there is no mpiexec in the mpich module I loaded, which is very weird. So, I reinstalled everything with openmpi. Now the make in tests gives the following error:

Linking Executable /work/wxia/share/software/MOOSE/projects/moose/test/moose_test-opt...
/usr/bin/ld: warning: libmpi.so.40, needed by /work/wxia/share/software/MOOSE/projects/moose/test/lib/libmoose_test-opt.so, may conflict with libmpi.so.12
/usr/bin/ld: warning: libmpi.so.40, needed by /work/wxia/share/software/MOOSE/projects/moose/test/lib/libmoose_test-opt.so, may conflict with libmpi.so.12
/usr/bin/ld: /work/wxia/share/software/MOOSE/projects/moose/test/src/.libs/main.x86_64-pc-linux-gnu.opt.o: undefined reference to symbol 'ompi_mpi_comm_world'
/usr/bin/ld: /opt/rit/el9/20230413/app/linux-rhel9-x86_64_v3/gcc-11.2.1/openmpi-4.1.5-mgignetyjqjz2etjbwh43qucpjmovesj/lib/libmpi.so.40: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make: *** [/work/wxia/share/software/MOOSE/projects/moose/framework/app.mk:466: /work/wxia/share/software/MOOSE/projects/moose/test/moose_test-opt] Error 1

[GiudGiud]

try mpirun then. mpiexec isnt provided by all distributionsmaybe

These errors means you are not using the same MPI distribution / set of compilers as the last time you compiled, or as when you compiled libmesh