Error in multi-process simulation

[CosimoMV]

Hello! I'm trying to install OpenParEm 3D, (a very interesting program and thank you @M8kmyday for making it open-source!), but I'm having problems.
I'm on Ubuntu 24.04, and I have compiled all the necessary packages following the guide on the pdf. But when I try to install Openparem2D and OpenPaEM3D I get errors in the compilation, which I can fix by using the ‘-fopenmp’ option in the compiler.
Once I go to test if everything works, using the command “process3D.sh straight.proj 2”, I get the reported error (I have correctly set export OMP_NUM_THREADS=1 in my bashrc file and this is not a RAM problem), while using a single process (process process3D.sh straight.proj 1) I can finish the simulation without errors. What am I doing wrong?
Thank you very much for your help.

      Driving S-port 1 ...
         solving E field in 3D volume ...
            using Cholesky preconditioner ...
*** buffer overflow detected ***: terminated
[cosimo-H610M:193042] *** Process received signal ***
[cosimo-H610M:193042] Signal: Aborted (6)
[cosimo-H610M:193042] Signal code:  (-6)
[cosimo-H610M:193042] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x45320)[0x747864c45320]
[cosimo-H610M:193042] [ 1] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x11c)[0x747864c9eb1c]
[cosimo-H610M:193042] [ 2] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x1e)[0x747864c4526e]
[cosimo-H610M:193042] [ 3] /lib/x86_64-linux-gnu/libc.so.6(abort+0xdf)[0x747864c288ff]
[cosimo-H610M:193042] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x297b6)[0x747864c297b6]
[cosimo-H610M:193042] [ 5] /lib/x86_64-linux-gnu/libc.so.6(+0x136c19)[0x747864d36c19]
[cosimo-H610M:193042] [ 6] /lib/x86_64-linux-gnu/libc.so.6(+0x1365d4)[0x747864d365d4]
[cosimo-H610M:193042] [ 7] OpenParEM3D(+0x161ad70)[0x5ba8aed1cd70]
[cosimo-H610M:193042] [ 8] OpenParEM3D(+0x161bc9f)[0x5ba8aed1dc9f]
[cosimo-H610M:193042] [ 9] OpenParEM3D(+0x161bc40)[0x5ba8aed1dc40]
[cosimo-H610M:193042] [10] OpenParEM3D(+0x161bc40)[0x5ba8aed1dc40]
[cosimo-H610M:193042] [11] OpenParEM3D(+0x161bc40)[0x5ba8aed1dc40]
[cosimo-H610M:193042] [12] OpenParEM3D(+0x161bf88)[0x5ba8aed1df88]
[cosimo-H610M:193042] [13] OpenParEM3D(+0x15f3e0e)[0x5ba8aecf5e0e]
[cosimo-H610M:193042] [14] OpenParEM3D(+0x15f3f4c)[0x5ba8aecf5f4c]
[cosimo-H610M:193042] [15] OpenParEM3D(+0x16090ba)[0x5ba8aed0b0ba]
[cosimo-H610M:193042] [16] OpenParEM3D(+0x1607830)[0x5ba8aed09830]
[cosimo-H610M:193042] [17] OpenParEM3D(+0x15ed8e9)[0x5ba8aecef8e9]
[cosimo-H610M:193042] [18] OpenParEM3D(+0x15edccc)[0x5ba8aecefccc]
[cosimo-H610M:193042] [19] OpenParEM3D(+0x15c0c5d)[0x5ba8aecc2c5d]
[cosimo-H610M:193042] [20] OpenParEM3D(+0x15c0d0e)[0x5ba8aecc2d0e]
[cosimo-H610M:193042] [21] OpenParEM3D(+0x15c06f9)[0x5ba8aecc26f9]
[cosimo-H610M:193042] [22] OpenParEM3D(+0x14c03d9)[0x5ba8aebc23d9]
[cosimo-H610M:193042] [23] OpenParEM3D(+0x1477d8b)[0x5ba8aeb79d8b]
[cosimo-H610M:193042] [24] OpenParEM3D(+0x13e096e)[0x5ba8aeae296e]
[cosimo-H610M:193042] [25] OpenParEM3D(+0x133c745)[0x5ba8aea3e745]
[cosimo-H610M:193042] [26] OpenParEM3D(+0x12dfeb3)[0x5ba8ae9e1eb3]
[cosimo-H610M:193042] [27] OpenParEM3D(+0x1267728)[0x5ba8ae969728]
[cosimo-H610M:193042] [28] OpenParEM3D(+0x1262f68)[0x5ba8ae964f68]
[cosimo-H610M:193042] [29] OpenParEM3D(+0xdaabb3)[0x5ba8ae4acbb3]
[cosimo-H610M:193042] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node cosimo-H610M exited on signal 6 (Aborted).
--------------------------------------------------------------------------
ERROR: File "straight_results.csv" is missing.

[M8kmyday]

Hi CosimoMV,

Thanks for giving OpenParEM3D a try.

There may be a problem with the compilation for parallel processing. OpenParEM3D is based on OpenMPI, and you mention OpenMP, which is not the same thing. You have success running in serial (np=1), where MPI is not invoked, then you have trouble with parallel execution (np>1). Perhaps in compiling, you are linking to OpenMP instead of OpenMPI? Although, I don't know if that is possible.

Have you tried running the pre-compiled binary? It is statically linked, so OpenMPI vs. OpenMP issues would not occur. If you can run the pre-compiled binary in parallel, then that would potentially point to linking to OpenMP when compiling.

[M8kmyday]

I installed Ubuntu 24.04.1 LTS in VirtualBox, and the pre-compiled binaries from 20.04.06 LTS did not run. I'll dig into it and provide an update when I have things running in 24.04.1 LTS.

[M8kmyday]

I get a clean compile in Ubuntu 24.04.1 LTS if I include one more library in section 5.1 of the installation guide (i.e. before compiling hypre):
sudo apt-get install libopenmpi-dev
I'll be updating the installation guide with that.

After that, I do see the behavior of a successful run in serial and fail with libc.so.6 in parallel. Working on that now.

[M8kmyday]

The pre-compiled binaries run in Ubuntu 22.04.5 LTS after adding a library [installation guide to be updated]:
sudo apt-get install libgfortran-10-dev
Both serial and parallel run without issues.

Ubuntu 24.04.1 LTS is having a problem with a PETSc call to solve the linear system. The problem persists even after updateing PETSc and SLEPc to their latest versions.

[CosimoMV]

Wow, thank you so much for your help!

On Ubuntu 24.04 I have already installed libopenmpi-dev. I also tried reinstalling all the packages linked to openMPI, but nothing changed. The thing about -fopenmp is really strange.

Is there any test I can do to help you debug OpenParEM3D on Ubuntu 24.04?

This is the error I get during compilation, if I don't set -fopenmp:

/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(memory.o): in function `get_num_procs':
(.text+0x1bc): undefined reference to `omp_get_num_places'
/usr/bin/ld: (.text+0x1d6): undefined reference to `omp_get_place_num_procs'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(dgemm.o): in function `dgemm_':
(.text+0x451): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x462): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(blas_server.o): in function `exec_threads':
(.text+0x22c): undefined reference to `omp_get_thread_num'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(blas_server.o): in function `exec_blas._omp_fn.0':
(.text+0x429): undefined reference to `omp_get_num_threads'
/usr/bin/ld: (.text+0x431): undefined reference to `omp_get_thread_num'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(blas_server.o): in function `exec_blas._omp_fn.1':
(.text+0x4c9): undefined reference to `omp_get_num_threads'
/usr/bin/ld: (.text+0x4d1): undefined reference to `omp_get_thread_num'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(blas_server.o): in function `blas_thread_init':
(.text+0x599): undefined reference to `omp_get_max_threads'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(blas_server.o): in function `exec_blas':
(.text+0x6b8): undefined reference to `GOMP_parallel'
/usr/bin/ld: (.text+0x6ed): undefined reference to `GOMP_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(ddot_k_NEHALEM.o): in function `ddot_k_NEHALEM':
(.text+0x347): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x362): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(ddot_k_SANDYBRIDGE.o): in function `ddot_k_SANDYBRIDGE':
(.text+0x357): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x372): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(zdotc_k_SANDYBRIDGE.o): in function `zdotc_k_SANDYBRIDGE':
(.text+0x4f7): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x512): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(zdotu_k_SANDYBRIDGE.o): in function `zdotu_k_SANDYBRIDGE':
(.text+0x4f7): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x512): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(zdotc_k_BULLDOZER.o): in function `zdotc_k_BULLDOZER':
(.text+0x487): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4a2): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(zdotu_k_BULLDOZER.o): in function `zdotu_k_BULLDOZER':
(.text+0x487): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4a2): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(ddot_k_PILEDRIVER.o): in function `ddot_k_PILEDRIVER':
(.text+0x437): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x452): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(zdotc_k_PILEDRIVER.o): in function `zdotc_k_PILEDRIVER':
(.text+0x467): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x482): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(zdotu_k_PILEDRIVER.o): in function `zdotu_k_PILEDRIVER':
(.text+0x467): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x482): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(ddot_k_STEAMROLLER.o): in function `ddot_k_STEAMROLLER':
(.text+0x357): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x372): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(zdotc_k_STEAMROLLER.o): in function `zdotc_k_STEAMROLLER':
(.text+0x467): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x482): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(zdotu_k_STEAMROLLER.o): in function `zdotu_k_STEAMROLLER':
(.text+0x467): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x482): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(ddot_k_EXCAVATOR.o): in function `ddot_k_EXCAVATOR':
(.text+0x357): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x372): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(zdotc_k_EXCAVATOR.o): in function `zdotc_k_EXCAVATOR':
(.text+0x467): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x482): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(zdotu_k_EXCAVATOR.o): in function `zdotu_k_EXCAVATOR':
(.text+0x467): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x482): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(dasum_k_HASWELL.o): in function `dasum_k_HASWELL':
(.text+0x2e1): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x2f1): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(ddot_k_HASWELL.o): in function `ddot_k_HASWELL':
(.text+0x347): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x362): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(drot_k_HASWELL.o): in function `drot_k_HASWELL':
(.text+0x2a3): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x2c2): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(zdotc_k_HASWELL.o): in function `zdotc_k_HASWELL':
(.text+0x4b7): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4d2): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(zdotu_k_HASWELL.o): in function `zdotu_k_HASWELL':
(.text+0x4b7): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4d2): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(ddot_k_ZEN.o): in function `ddot_k_ZEN':
(.text+0x347): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x362): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(drot_k_ZEN.o): in function `drot_k_ZEN':
(.text+0x2a3): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x2c2): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(zdotc_k_ZEN.o): in function `zdotc_k_ZEN':
(.text+0x4b7): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4d2): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(zdotu_k_ZEN.o): in function `zdotu_k_ZEN':
(.text+0x4b7): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4d2): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(dasum_k_SKYLAKEX.o): in function `dasum_k_SKYLAKEX':
(.text+0x321): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x331): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(ddot_k_SKYLAKEX.o): in function `ddot_k_SKYLAKEX':
(.text+0x3d6): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x3f8): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(drot_k_SKYLAKEX.o): in function `drot_k_SKYLAKEX':
(.text+0x2ff): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x322): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(casum_k_SKYLAKEX.o): in function `casum_k_SKYLAKEX':
(.text+0x771): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x781): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(zasum_k_SKYLAKEX.o): in function `zasum_k_SKYLAKEX':
(.text+0x6f1): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x701): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(zdotc_k_SKYLAKEX.o): in function `zdotc_k_SKYLAKEX':
(.text+0x496): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4b8): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(zdotu_k_SKYLAKEX.o): in function `zdotu_k_SKYLAKEX':
(.text+0x496): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4b8): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(dasum_k_COOPERLAKE.o): in function `dasum_k_COOPERLAKE':
(.text+0x321): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x331): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(ddot_k_COOPERLAKE.o): in function `ddot_k_COOPERLAKE':
(.text+0x3d6): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x3f8): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(drot_k_COOPERLAKE.o): in function `drot_k_COOPERLAKE':
(.text+0x31f): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x342): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(casum_k_COOPERLAKE.o): in function `casum_k_COOPERLAKE':
(.text+0x771): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x781): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(zasum_k_COOPERLAKE.o): in function `zasum_k_COOPERLAKE':
(.text+0x6f1): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x701): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(zdotc_k_COOPERLAKE.o): in function `zdotc_k_COOPERLAKE':
(.text+0x496): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4b8): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(zdotu_k_COOPERLAKE.o): in function `zdotu_k_COOPERLAKE':
(.text+0x496): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4b8): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(dasum_k_SAPPHIRERAPIDS.o): in function `dasum_k_SAPPHIRERAPIDS':
(.text+0x321): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x331): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(ddot_k_SAPPHIRERAPIDS.o): in function `ddot_k_SAPPHIRERAPIDS':
(.text+0x3b6): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x3d8): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(drot_k_SAPPHIRERAPIDS.o): in function `drot_k_SAPPHIRERAPIDS':
(.text+0x31f): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x342): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(casum_k_SAPPHIRERAPIDS.o): in function `casum_k_SAPPHIRERAPIDS':
(.text+0x771): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x781): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(zasum_k_SAPPHIRERAPIDS.o): in function `zasum_k_SAPPHIRERAPIDS':
(.text+0x6f1): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x701): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(zdotc_k_SAPPHIRERAPIDS.o): in function `zdotc_k_SAPPHIRERAPIDS':
(.text+0x496): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4b8): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(zdotu_k_SAPPHIRERAPIDS.o): in function `zdotu_k_SAPPHIRERAPIDS':
(.text+0x496): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4b8): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(daxpy.o): in function `daxpy_':
(.text+0xa4): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0xbd): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(sgemm_direct_performant_PRESCOTT.o): in function `sgemm_direct_performant_PRESCOTT':
(.text+0x42): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4e): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(sgemm_direct_performant_CORE2.o): in function `sgemm_direct_performant_CORE2':
(.text+0x42): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4e): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(sgemm_direct_performant_PENRYN.o): in function `sgemm_direct_performant_PENRYN':
(.text+0x42): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4e): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(sgemm_direct_performant_DUNNINGTON.o): in function `sgemm_direct_performant_DUNNINGTON':
(.text+0x42): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4e): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(sgemm_direct_performant_NEHALEM.o): in function `sgemm_direct_performant_NEHALEM':
(.text+0x42): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4e): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(sgemm_direct_performant_OPTERON.o): in function `sgemm_direct_performant_OPTERON':
(.text+0x42): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4e): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(sgemm_direct_performant_OPTERON_SSE3.o): in function `sgemm_direct_performant_OPTERON_SSE3':
(.text+0x42): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4e): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(sgemm_direct_performant_BARCELONA.o): in function `sgemm_direct_performant_BARCELONA':
(.text+0x42): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4e): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(sgemm_direct_performant_BOBCAT.o): in function `sgemm_direct_performant_BOBCAT':
(.text+0x42): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4e): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(sgemm_direct_performant_ATOM.o): in function `sgemm_direct_performant_ATOM':
(.text+0x42): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4e): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(sgemm_direct_performant_NANO.o): in function `sgemm_direct_performant_NANO':
(.text+0x42): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4e): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(sgemm_direct_performant_SANDYBRIDGE.o): in function `sgemm_direct_performant_SANDYBRIDGE':
(.text+0x42): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4e): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(sgemm_direct_performant_BULLDOZER.o): in function `sgemm_direct_performant_BULLDOZER':
(.text+0x42): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4e): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(sgemm_direct_performant_PILEDRIVER.o): in function `sgemm_direct_performant_PILEDRIVER':
(.text+0x42): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4e): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(sgemm_direct_performant_STEAMROLLER.o): in function `sgemm_direct_performant_STEAMROLLER':
(.text+0x42): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4e): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(sgemm_direct_performant_EXCAVATOR.o): in function `sgemm_direct_performant_EXCAVATOR':
(.text+0x42): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4e): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(sasum_k_HASWELL.o): in function `sasum_k_HASWELL':
(.text+0x2d1): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x2e1): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(srot_k_HASWELL.o): in function `srot_k_HASWELL':
(.text+0x2a3): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x2c2): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(sgemm_direct_performant_HASWELL.o): in function `sgemm_direct_performant_HASWELL':
(.text+0x42): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4e): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(srot_k_ZEN.o): in function `srot_k_ZEN':
(.text+0x2a3): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x2c2): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(sgemm_direct_performant_ZEN.o): in function `sgemm_direct_performant_ZEN':
(.text+0x42): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4e): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(sasum_k_SKYLAKEX.o): in function `sasum_k_SKYLAKEX':
(.text+0x301): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x311): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(srot_k_SKYLAKEX.o): in function `srot_k_SKYLAKEX':
(.text+0x2ff): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x322): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(sgemm_direct_performant_SKYLAKEX.o): in function `sgemm_direct_performant_SKYLAKEX':
(.text+0x42): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4e): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(sasum_k_COOPERLAKE.o): in function `sasum_k_COOPERLAKE':
(.text+0x301): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x311): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(srot_k_COOPERLAKE.o): in function `srot_k_COOPERLAKE':
(.text+0x31f): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x342): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(sgemm_direct_performant_COOPERLAKE.o): in function `sgemm_direct_performant_COOPERLAKE':
(.text+0x42): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4e): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(sasum_k_SAPPHIRERAPIDS.o): in function `sasum_k_SAPPHIRERAPIDS':
(.text+0x301): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x311): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(srot_k_SAPPHIRERAPIDS.o): in function `srot_k_SAPPHIRERAPIDS':
(.text+0x31f): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x342): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(sgemm_direct_performant_SAPPHIRERAPIDS.o): in function `sgemm_direct_performant_SAPPHIRERAPIDS':
(.text+0x42): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x4e): undefined reference to `omp_in_parallel'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/liblapack64.a(dgemv.o): in function `dgemv_':
(.text+0x398): undefined reference to `omp_get_max_threads'
/usr/bin/ld: (.text+0x3c3): undefined reference to `omp_in_parallel'
collect2: error: ld returned 1 exit status
make: *** [makefile:75: OpenParEM2D] Errore 1

[M8kmyday]

I'm thinking that the -fopenmp flag is ok. Your error messages are essentially referencing Intel processors, so I think you have Intel-optimized libraries installed and would require some additional information to the compiler to properly utilize these.

Another reason why I'm thinking that the -fopenmp flag is ok is that I'm seeing the same behavior that you are seeing: proper execution in serial and the same error in parallel. Also note that OpenParEM2D runs properly in parallel for the failing case before it gets to the error, so parallel processing is fundamentally working even for the failing case.

To summarize on the error, it does not occur for Ubuntu 20.04.06 LTS or Ubuntu 22.04.05 LTS. We both see the error in Ubuntu 24.04.1 LTS. The error occurs at a PETSc call to KSPSolve. I have tried updating PETSc and SLEPc to the latest versions and recompiling, but that does not resolve the error.

The next step is to investigate the latest or an earlier version of libc, which is the library reporting the error. After that, I'll try Valgrind to look for a memory problem in OpenParEM. [Note that Valgrind has been extensively used on OpenParEM in Ubuntu 20.04.06 LTS, but maybe something pops up in Ubuntu 24.04.1 LTS?]

If you want to try any of these ideas or something else, that would be great. I'll keep working on my side because it would be great to be able to support Ubuntu 24.

[M8kmyday]

This is looking like a complex issue between PETSc and Ubuntu 24.04.1 LTS. The errors with libc.so.6 do not appear if the pre-conditioner to the KSPSolve is changed from PCCHOLESKY to PCJACOBI or PCNONE, a simple parameter change. [However, convergence is not achieved with these options.] From the perspective of OpenParEM3D, successfully running (if not converging) with one of these changes somewhat rules out a fix based in OpenParEM code.

Running Valgrind did not turn up anything new, and compiling and installing the latest version of OpenMPI did not affect the error. Already tried was ensuring that Ubuntu 24.04.1 LTS is fully up-to-date and that PETSc is using the latest version. libc is very fundamental to Linux, so I decided to not mess with libc.

Given that Ubuntu 24.04.1 LTS just came out in late April, perhaps it and the infrastructure are not yet quite ready?

I will fully check out Ubuntu 22 by verifying it with the regression suite and updating the documentation. Users will then have the option of Ubuntu 20 and 22, and I think Ubuntu 24 will have to wait for another day.

[M8kmyday]

Ubuntu 22.04.05 LTS passed the regression suites, so it is good for use with OpenParEM. I will update documentation and release that, then I will re-visit Ubuntu 24.

[CosimoMV]

I tried compiling glibc 2.40 and got some errors, so I tuned the compiler options to remove them (I just googled the error) and tried recompiling OpenParem with these flags (on glibc 2.39), but nothing changed. I suspected a change in glibc from Ubunutn 22.04 and Ubunutnu 24, and I found that from glibc 2.34 libpthread was removed as a separate library... but Ubuntu 22.04 has glibc 2.35, so this can't be the problem.

[M8kmyday]

Thanks for giving that a shot. My current plan is to turn on debugging in PETSc, then trace back the failing point to the specific PETSc code. That effort may suggest a fix or enable a detailed bug report to the PETSc or Ubuntu teams. I don't think this is going to be a quick fix.

[M8kmyday]

I found the problem, so it didn't take as long I feared. A change in 6 lines of code in OpenParEM3D allows Ubuntu 24 to work. I need to audit the code to see if there are similar changes required elsewhere then re-run the regression suites across Ubuntu 20, 22, and 24, then re-release. This will take a few days due to run times on the regression suites. If you do not want to wait, I can tell you the 6 lines to change so that you can be running while I to the work on a formal release.

[CosimoMV]

If you do not want to wait, I can tell you the 6 lines to change so that you can be running while I to the work on a formal release.

Yes, I can't wait to try it! Please tell me if there is any particular test you would like me to try. To run all the tests in the regression suites, is the command the documented one in the guide?

[M8kmyday]

I pushed the changes to OpenParEM3D last night, so you can pull main to get the latest. I have not created a release for this version yet.

In the OpenParEM3D regression directory, check the README.txt file. Step 4 shows how to run entire regression suite. The script assumes that you have 12+ cores on your computer, so if you do not, you will want to reduce the number of cores in the file regression_case_list.txt to avoid overloading the computer and slowing things down.

There is a simple script called check.sh that checks the results for problems, and you can run this at any time during the run to see how things are going. No output is good. Or, you can manually check regression.log and regression_results.csv.

[CosimoMV]

I tried recompiling OpenParEM3D again with the latest changes, but I still get errors at run time if I use more than one process.
On Monday I will try again by compiling everything from scratch.

This is the error I get with process3D.sh straight.proj 2:

      building finite element spaces ...
      Driving S-port 1 ...
         solving E field in 3D volume ...
            using Cholesky preconditioner ...
*** buffer overflow detected ***: terminated
[cosimo-H610M:163945] *** Process received signal ***
[cosimo-H610M:163945] Signal: Aborted (6)
[cosimo-H610M:163945] Signal code:  (-6)
[cosimo-H610M:163945] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x45320)[0x75cba4245320]
[cosimo-H610M:163945] [ 1] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x11c)[0x75cba429eb1c]
[cosimo-H610M:163945] [ 2] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x1e)[0x75cba424526e]
[cosimo-H610M:163945] [ 3] /lib/x86_64-linux-gnu/libc.so.6(abort+0xdf)[0x75cba42288ff]
[cosimo-H610M:163945] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x297b6)[0x75cba42297b6]
[cosimo-H610M:163945] [ 5] /lib/x86_64-linux-gnu/libc.so.6(+0x136c19)[0x75cba4336c19]
[cosimo-H610M:163945] [ 6] /lib/x86_64-linux-gnu/libc.so.6(+0x1365d4)[0x75cba43365d4]
[cosimo-H610M:163945] [ 7] OpenParEM3D(+0x1629b20)[0x5ccb87e31b20]
[cosimo-H610M:163945] [ 8] OpenParEM3D(+0x162aa4f)[0x5ccb87e32a4f]
[cosimo-H610M:163945] [ 9] OpenParEM3D(+0x162a9f0)[0x5ccb87e329f0]
[cosimo-H610M:163945] [10] OpenParEM3D(+0x162a9f0)[0x5ccb87e329f0]
[cosimo-H610M:163945] [11] OpenParEM3D(+0x162a9f0)[0x5ccb87e329f0]
[cosimo-H610M:163945] [12] OpenParEM3D(+0x162ad38)[0x5ccb87e32d38]
[cosimo-H610M:163945] [13] OpenParEM3D(+0x1602bbe)[0x5ccb87e0abbe]
[cosimo-H610M:163945] [14] OpenParEM3D(+0x1602cfc)[0x5ccb87e0acfc]
[cosimo-H610M:163945] [15] OpenParEM3D(+0x1617e6a)[0x5ccb87e1fe6a]
[cosimo-H610M:163945] [16] OpenParEM3D(+0x16165e0)[0x5ccb87e1e5e0]
[cosimo-H610M:163945] [17] OpenParEM3D(+0x15fc699)[0x5ccb87e04699]
[cosimo-H610M:163945] [18] OpenParEM3D(+0x15fca7c)[0x5ccb87e04a7c]
[cosimo-H610M:163945] [19] OpenParEM3D(+0x15cfa0d)[0x5ccb87dd7a0d]
[cosimo-H610M:163945] [20] OpenParEM3D(+0x15cfabe)[0x5ccb87dd7abe]
[cosimo-H610M:163945] [21] OpenParEM3D(+0x15cf4a9)[0x5ccb87dd74a9]
[cosimo-H610M:163945] [22] OpenParEM3D(+0x14cf189)[0x5ccb87cd7189]
[cosimo-H610M:163945] [23] OpenParEM3D(+0x1486b3b)[0x5ccb87c8eb3b]
[cosimo-H610M:163945] [24] OpenParEM3D(+0x13ef71e)[0x5ccb87bf771e]
[cosimo-H610M:163945] [25] OpenParEM3D(+0x134b4f5)[0x5ccb87b534f5]
[cosimo-H610M:163945] [26] OpenParEM3D(+0x12eec63)[0x5ccb87af6c63]
[cosimo-H610M:163945] [27] OpenParEM3D(+0x12764d8)[0x5ccb87a7e4d8]
[cosimo-H610M:163945] [28] OpenParEM3D(+0x1271d18)[0x5ccb87a79d18]
[cosimo-H610M:163945] [29] OpenParEM3D(+0xdb9963)[0x5ccb875c1963]
[cosimo-H610M:163945] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node cosimo-H610M exited on signal 6 (Aborted).
--------------------------------------------------------------------------

[M8kmyday]

Yes, that is the same error we were both seeing before. After I made the change to fem3D.cpp, the problem was resolved. I'm thinking that you somehow did not pick up the change. In fem3D.cpp, you should see lines 455-458 look like this:
455 // clean up
456 PetscFree(ixvals);
457 PetscFree(local_x_vals);
458 PetscFree(global_x_vals);
If not, then you still have the old code.

[CosimoMV]

I checked fem3D.cpp to be sure and I have the new version, but I still get the error.

[M8kmyday]

Well, that is frustrating.

Here is a fully detailed experiment I just did using pre-compiled binaries, and it works under Ubuntu 24.04.1 LTS running in VirtualBox. By trying the binaries, all compilation variables are eliminated.

Remove existing executables:

1. which OpenParEM2D
2. rm the OpenParEM2D that is reported
3. which OpenParEM2D
4. verify that nothing is found
5. which OpenParEM3D
6. rm the OpenParEM3D that is reported
7. which OpenParEM3D
8. verify that nothing is found

Download precompiled binaries:

1. Navigate to the bin directory of branch 'main' in the OpenParEM2D repository on GitHub
2. Click on "OpenParEM2D"
3. Click the "..." on the right
4. Click download
5. Move the downloaded file from the Downloads directory to $HOME/bin or to the executable directory you are using
6. Navigate to the bin directory of branch 'main' in the OpenParEM3D repository on GitHub
7. Click on "OpenParEM3D"
8. Click the "..." on the right
9. Click download
10. Move the downloaded file from the Downloads directory to $HOME/bin or to the executable directory you are using
11. cd to the executable directory
12. chmod u+x OpenParEM2D
13. chmod u+x OpenParEM3D

Check:

1. cd OpenParEM3D/regression/WR90/straight
2. which OpenParEM2D
3. verify that it is found
4. which OpenParEM3D
5. verify that it is found

Run:

1. mpirun -np 2 OpenParEM3D straight.proj
2. See if it runs or reports an error

If this works, then the problem is somewhere in the compilation process. If it doesn't work, then there is a difference in our Ubuntu 24 installations or VirtualBox is somehow papering over a problem.

[CosimoMV]

I have just performed the waveguide test with the pre-compiled binaries, and now it works! So it is my fault during the installation procedure.

Just for my knowledge, could you tell me the packages versions and variables you used during compilation? I followed the procedure on the pdf step by step, with the only change being the variable ‘-fopenmp’ for compiling OpenParEm2D and OpenParEM3D (otherwise the procedure would give an error).
My packages versions:

Package	Version
Hypre	2.31.0
Metis	4.0.3
Mfem	4.7
Petsc	3.21.5
Slepc	3.21.1

[M8kmyday]

Great news! I'm glad you have something working now.

The versions are called out in the installation manual, and here is a recap:

Hypre 2.30.0
Metis 4.0.3
MFEM 4.7
Petsc 3.19.6
Slepc 3.19.2

I had actually tried upgrading to the latest Petsc and Slepc, and it had no effect, so I doubt that that is the source of your issue. I doubt Hypre is the issue since the failure occurs in Petsc and it does not use Hypre. The only thing I can think of right now is ensuring that your Ubuntu 24 is fully updated, which mine is since I just installed it. Good luck!