Nightly petsc build unable to load libscotcherr-6.so

[guyer]

• Nightly build has been failing for petsc due to
  ImportError: Unable to load solver petsc: libscotcherr-6.so: cannot open shared object file: No such file or directory

• linux and macOS builds both fail

• Difference from PR builds is that
  daily midnight build uses environment.yml files to catch regressions
  due to updates in dependencies

• Nightly build installs

  - python=3.7.12=hb7a2778_100_cpython
  - petsc=3.10.3=blas_openblashe91a55c_1000
  - petsc4py=3.10.0=py37h0587ea6_1003

• Most recent successful PR build installs

  - python=3.12.3=hab00c5b_0_cpython
  - petsc=3.20.2=real_h622a54c_100
  - petsc4py=3.20.2=real_h098918d_100

[guyer]

Attempt to replicate environment on local macOS machine:

conda env create --name libscotcherr-6.so --file environments/petsc

$ conda list
# packages in environment at /Users/guyer/mambaforge-arm/envs/libscotcherr-6.so:
#
# Name                    Version                   Build  Channel
aiohttp                   3.9.5           py312he37b823_0    conda-forge
aiosignal                 1.3.1              pyhd8ed1ab_0    conda-forge
aom                       3.9.1                h7bae524_0    conda-forge
apptools                  5.1.0              pyh44b312d_0    conda-forge
attrs                     23.2.0             pyh71513ae_0    conda-forge
blosc                     1.21.5               h9c252e8_1    conda-forge
brotli                    1.1.0                hb547adb_1    conda-forge
brotli-bin                1.1.0                hb547adb_1    conda-forge
bzip2                     1.0.8                h99b78c6_7    conda-forge
c-ares                    1.32.3               h99b78c6_0    conda-forge
ca-certificates           2024.7.4             hf0a4a13_0    conda-forge
cairo                     1.18.0               hd1e100b_0    conda-forge
certifi                   2024.7.4           pyhd8ed1ab_0    conda-forge
configobj                 5.0.8              pyhd8ed1ab_0    conda-forge
contourpy                 1.2.1           py312h0fef576_0    conda-forge
cycler                    0.12.1             pyhd8ed1ab_0    conda-forge
dav1d                     1.2.1                hb547adb_0    conda-forge
double-conversion         3.3.0                h13dd4ca_0    conda-forge
eigen                     3.4.0                h1995070_0    conda-forge
envisage                  6.0.1              pyhd8ed1ab_0    conda-forge
expat                     2.6.2                hebf3989_0    conda-forge
ffmpeg                    6.1.1           gpl_hc16618e_112    conda-forge
fftw                      3.3.10          mpi_mpich_hd148bab_10    conda-forge
fltk                      1.3.9                h31b9a01_0    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 h77eed37_2    conda-forge
fontconfig                2.14.2               h82840c6_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.53.1          py312h7e5086c_0    conda-forge
freeimage                 3.18.0              hd0e3f39_20    conda-forge
freetype                  2.12.1               hadb7bae_2    conda-forge
fribidi                   1.0.10               h27ca646_0    conda-forge
frozenlist                1.4.1           py312he37b823_0    conda-forge
future                    1.0.0              pyhd8ed1ab_0    conda-forge
gettext                   0.22.5               h8fbad5d_2    conda-forge
gettext-tools             0.22.5               h8fbad5d_2    conda-forge
gl2ps                     1.4.2                h17b34a0_0    conda-forge
glew                      2.1.0                h9f76cd9_2    conda-forge
glib                      2.80.2               h535f939_0    conda-forge
glib-tools                2.80.2               h4c882b9_0    conda-forge
gmp                       6.3.0                h7bae524_2    conda-forge
gmsh                      4.12.2               hd427cfb_0    conda-forge
gnutls                    3.7.9                hd26332c_0    conda-forge
graphite2                 1.3.13            hebf3989_1003    conda-forge
gst-plugins-base          1.24.4               h8a8f8c8_0    conda-forge
gstreamer                 1.24.4               h430e707_0    conda-forge
harfbuzz                  8.5.0                h1836168_0    conda-forge
hdf4                      4.2.15               h2ee6834_7    conda-forge
hdf5                      1.14.3          mpi_mpich_hb65fe4a_5    conda-forge
hypre                     2.28.0          mpi_mpich_hd226f01_0    conda-forge
icu                       73.2                 hc8870d7_0    conda-forge
idna                      3.7                pyhd8ed1ab_0    conda-forge
imath                     3.1.11               h1059232_0    conda-forge
importlib-metadata        8.2.0              pyha770c72_0    conda-forge
importlib_metadata        8.2.0                hd8ed1ab_0    conda-forge
importlib_resources       6.4.0              pyhd8ed1ab_0    conda-forge
jsoncpp                   1.9.5                hc021e02_1    conda-forge
jxrlib                    1.1                  h93a5062_3    conda-forge
kiwisolver                1.4.5           py312h389731b_1    conda-forge
krb5                      1.21.3               h237132a_0    conda-forge
lame                      3.100             h1a8c8d9_1003    conda-forge
lcms2                     2.16                 ha0e7c42_0    conda-forge
lerc                      4.0.0                h9a09cb3_0    conda-forge
libabseil                 20240116.2      cxx17_h00cdb27_1    conda-forge
libaec                    1.1.3                hebf3989_0    conda-forge
libasprintf               0.22.5               h8fbad5d_2    conda-forge
libasprintf-devel         0.22.5               h8fbad5d_2    conda-forge
libass                    0.17.1               hf7da4fe_1    conda-forge
libblas                   3.9.0           23_osxarm64_openblas    conda-forge
libbrotlicommon           1.1.0                hb547adb_1    conda-forge
libbrotlidec              1.1.0                hb547adb_1    conda-forge
libbrotlienc              1.1.0                hb547adb_1    conda-forge
libcblas                  3.9.0           23_osxarm64_openblas    conda-forge
libclang-cpp15            15.0.7          default_he012953_5    conda-forge
libclang-cpp16            16.0.6          default_h5c12605_11    conda-forge
libclang13                18.1.7          default_hb9c8b4a_0    conda-forge
libcurl                   8.8.0                h7b6f9a7_1    conda-forge
libcxx                    18.1.8               h167917d_0    conda-forge
libdeflate                1.20                 h93a5062_0    conda-forge
libedit                   3.1.20191231         hc8eb9b7_2    conda-forge
libev                     4.33                 h93a5062_2    conda-forge
libexpat                  2.6.2                hebf3989_0    conda-forge
libffi                    3.4.2                h3422bc3_5    conda-forge
libgettextpo              0.22.5               h8fbad5d_2    conda-forge
libgettextpo-devel        0.22.5               h8fbad5d_2    conda-forge
libgfortran               5.0.0           13_2_0_hd922786_3    conda-forge
libgfortran5              13.2.0               hf226fd6_3    conda-forge
libglib                   2.80.2               h535f939_0    conda-forge
libhwloc                  2.11.1          default_h7685b71_1000    conda-forge
libiconv                  1.17                 h0d3ecfb_2    conda-forge
libidn2                   2.3.7                h93a5062_0    conda-forge
libintl                   0.22.5               h8fbad5d_2    conda-forge
libintl-devel             0.22.5               h8fbad5d_2    conda-forge
libjpeg-turbo             3.0.0                hb547adb_1    conda-forge
liblapack                 3.9.0           23_osxarm64_openblas    conda-forge
libllvm15                 15.0.7               h2621b3d_4    conda-forge
libllvm16                 16.0.6               haab561b_3    conda-forge
libllvm18                 18.1.7               hdac5640_0    conda-forge
libnetcdf                 4.9.2           nompi_he469be0_114    conda-forge
libnghttp2                1.58.0               ha4dd798_1    conda-forge
libogg                    1.3.5                h99b78c6_0    conda-forge
libopenblas               0.3.27          openmp_h517c56d_1    conda-forge
libopenvino               2024.1.0             h5c9529b_7    conda-forge
libopenvino-arm-cpu-plugin 2024.1.0             h5c9529b_7    conda-forge
libopenvino-auto-batch-plugin 2024.1.0             hcd65546_7    conda-forge
libopenvino-auto-plugin   2024.1.0             hcd65546_7    conda-forge
libopenvino-hetero-plugin 2024.1.0             h88cb26a_7    conda-forge
libopenvino-ir-frontend   2024.1.0             h88cb26a_7    conda-forge
libopenvino-onnx-frontend 2024.1.0             h32b5460_7    conda-forge
libopenvino-paddle-frontend 2024.1.0             h32b5460_7    conda-forge
libopenvino-pytorch-frontend 2024.1.0             h00cdb27_7    conda-forge
libopenvino-tensorflow-frontend 2024.1.0             h2741c3b_7    conda-forge
libopenvino-tensorflow-lite-frontend 2024.1.0             h00cdb27_7    conda-forge
libopus                   1.3.1                h27ca646_1    conda-forge
libpng                    1.6.43               h091b4b1_0    conda-forge
libpq                     16.3                 h7afe498_0    conda-forge
libprotobuf               4.25.3               hbfab5d5_0    conda-forge
libraw                    0.21.1               h2ee6834_2    conda-forge
libsqlite                 3.46.0               hfb93653_0    conda-forge
libssh2                   1.11.0               h7a5bd25_0    conda-forge
libtasn1                  4.19.0               h1a8c8d9_0    conda-forge
libtheora                 1.1.1             h99b78c6_1006    conda-forge
libtiff                   4.6.0                h07db509_3    conda-forge
libunistring              0.9.10               h3422bc3_0    conda-forge
libvorbis                 1.3.7                h9f76cd9_0    conda-forge
libvpx                    1.14.1               h7bae524_0    conda-forge
libwebp-base              1.4.0                h93a5062_0    conda-forge
libxcb                    1.15                 hf346824_0    conda-forge
libxml2                   2.12.7               ha661575_1    conda-forge
libzip                    1.10.1               ha0bc3c6_3    conda-forge
libzlib                   1.2.13               hfb2fe0b_6    conda-forge
llvm-openmp               18.1.8               hde57baf_0    conda-forge
loguru                    0.7.2           py312h81bd7bf_1    conda-forge
lz4-c                     1.9.4                hb7217d7_0    conda-forge
matplotlib-base           3.9.1           py312h32d6e5a_0    conda-forge
mayavi                    4.8.2           pyqt_py312h6b0e786_204    conda-forge
metis                     5.1.0             h13dd4ca_1007    conda-forge
mpfr                      4.2.1                h41d338b_1    conda-forge
mpi                       1.0                       mpich    conda-forge
mpi4py                    3.1.6           py312h66645bf_1    conda-forge
mpich                     4.2.2              h9ee2227_100    conda-forge
msgpack-python            1.0.8           py312h157fec4_0    conda-forge
multidict                 6.0.5           py312h670c8ac_0    conda-forge
mumps-include             5.2.1               hce30654_11    conda-forge
mumps-mpi                 5.2.1               hfd295ca_11    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
mysql-common              8.3.0                hd1853d3_4    conda-forge
mysql-libs                8.3.0                hf036fc4_4    conda-forge
ncurses                   6.5                  hb89a1cb_0    conda-forge
nettle                    3.9.1                h40ed0f5_0    conda-forge
nlohmann_json             3.11.3               hebf3989_0    conda-forge
nspr                      4.35                 hb7217d7_0    conda-forge
nss                       3.100                hc6e9f88_0    conda-forge
numpy                     1.26.4          py312h8442bc7_0    conda-forge
occt                      7.7.2           novtk_h5f4376a_101    conda-forge
openexr                   3.2.2                h2c51e1d_1    conda-forge
openh264                  2.4.1                hebf3989_0    conda-forge
openjpeg                  2.5.2                h9f1df11_0    conda-forge
openssl                   3.3.1                hfb2fe0b_2    conda-forge
p11-kit                   0.24.1               h29577a5_0    conda-forge
packaging                 24.1               pyhd8ed1ab_0    conda-forge
parmetis                  4.0.3             hb315350_1006    conda-forge
pcre2                     10.43                h26f9a81_0    conda-forge
petsc                     3.20.2          real_hfe62963_100    conda-forge
petsc4py                  3.20.2          real_hd567e68_100    conda-forge
pillow                    10.3.0          py312h8a801b1_0    conda-forge
pip                       24.0               pyhd8ed1ab_0    conda-forge
pixman                    0.43.4               hebf3989_0    conda-forge
ply                       3.11               pyhd8ed1ab_2    conda-forge
proj                      9.3.1                h93d94ba_0    conda-forge
pthread-stubs             0.4               h27ca646_1001    conda-forge
ptscotch                  6.0.9                hd02db47_2    conda-forge
pugixml                   1.14                 h13dd4ca_0    conda-forge
pyface                    8.0.0              pyhd8ed1ab_1    conda-forge
pygments                  2.18.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.1.2              pyhd8ed1ab_0    conda-forge
pyqt                      5.15.9          py312h550cae4_5    conda-forge
pyqt5-sip                 12.12.2         py312h9f69965_5    conda-forge
python                    3.12.3          h4a7b5fc_0_cpython    conda-forge
python-dateutil           2.9.0              pyhd8ed1ab_0    conda-forge
python_abi                3.12                    4_cp312    conda-forge
qhull                     2020.2               h420ef59_5    conda-forge
qt-main                   5.15.8              hf679f28_21    conda-forge
qt6-main                  6.7.1                h845717a_2    conda-forge
rapidjson                 1.1.0.post20240409      h00cdb27_1    conda-forge
readline                  8.2                  h92ec313_1    conda-forge
scalapack                 2.2.0                h908211e_2    conda-forge
scipy                     1.14.0          py312h14ffa8f_1    conda-forge
scotch                    6.0.9                h7537618_2    conda-forge
setuptools                71.0.4             pyhd8ed1ab_0    conda-forge
sip                       6.8.3           py312h20a0b95_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.2.1                hd02b534_0    conda-forge
sqlite                    3.46.0               h5838104_0    conda-forge
suitesparse               5.10.1               h79486c6_3    conda-forge
superlu                   5.2.2                hc615359_0    conda-forge
superlu_dist              7.2.0                h7df07b8_0    conda-forge
svt-av1                   2.1.0                h7bae524_0    conda-forge
tbb                       2021.12.0            h420ef59_3    conda-forge
tbb-devel                 2021.12.0            h5309751_3    conda-forge
tk                        8.6.13               h5083fa2_1    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
traits                    6.4.3           py312he37b823_0    conda-forge
traitsui                  8.0.0              pyhd8ed1ab_0    conda-forge
tzdata                    2024a                h0c530f3_0    conda-forge
utfcpp                    4.0.5                hce30654_0    conda-forge
vtk                       9.3.0           qt_py312h1234567_200    conda-forge
vtk-base                  9.3.0           qt_py312h1234567_200    conda-forge
vtk-io-ffmpeg             9.3.0           qt_py312h1234567_200    conda-forge
wheel                     0.43.0             pyhd8ed1ab_1    conda-forge
wslink                    2.1.1              pyhd8ed1ab_0    conda-forge
x264                      1!164.3095           h57fd34a_2    conda-forge
x265                      3.5                  hbc6ce65_3    conda-forge
xorg-fixesproto           5.0               h3422bc3_1002    conda-forge
xorg-kbproto              1.0.7             h27ca646_1002    conda-forge
xorg-libice               1.1.1                hb547adb_0    conda-forge
xorg-libsm                1.2.4                hb547adb_0    conda-forge
xorg-libx11               1.8.9                h570a39f_0    conda-forge
xorg-libxau               1.0.11               hb547adb_0    conda-forge
xorg-libxdmcp             1.1.3                h27ca646_0    conda-forge
xorg-libxext              1.3.4                h1a8c8d9_2    conda-forge
xorg-libxfixes            5.0.3             h3422bc3_1004    conda-forge
xorg-libxrender           0.9.11               hb547adb_0    conda-forge
xorg-renderproto          0.11.1            h27ca646_1002    conda-forge
xorg-xextproto            7.3.0             h1a8c8d9_1003    conda-forge
xorg-xproto               7.0.31            h27ca646_1007    conda-forge
xz                        5.2.6                h57fd34a_0    conda-forge
yaml                      0.2.5                h3422bc3_2    conda-forge
yarl                      1.9.4           py312he37b823_0    conda-forge
zipp                      3.19.2             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               hfb2fe0b_6    conda-forge
zstd                      1.5.6                hb46c0d2_0    conda-forge

gets

python                    3.12.3          h4a7b5fc_0_cpython    conda-forge
petsc                     3.20.2          real_hfe62963_100    conda-forge
petsc4py                  3.20.2          real_hd567e68_100    conda-forge

so problem is something about Azure environment.

[guyer]

so problem is something about Azure environment.

No. Previous also reported

==> WARNING: A newer version of conda exists. <==                                                                                          
    current version: 23.7.4                                                                                                                
    latest version: 24.5.0

Please update conda by running

    $ conda update -n base -c conda-forge conda

Multiple attempts to update conda did not work, but did not error.

Finally, mamba install -n base -c conda-forge "conda>=24.5.0" worked

Important

Don't forget the quotation marks!!!

Subsequently, CONDA_SUBDIR=osx-64 conda env create --name libscotcherr-6.so-osx-64-bis-bis-bis-bis --file environments/petsc-environment.yml installs

python                    3.7.12          hf3644f1_100_cpython    conda-forge
petsc                     3.10.3          blas_openblasha70941c_1000    conda-forge
petsc4py                  3.10.0          py37h9e70694_1003    conda-forge

so my laptop is broken, now, but at least I can debug...

[guyer]

In contrast, conda env create --name libscotcherr-6.so-osx-64-bis-bis-bis-bis-bis --file environments/petsc-environment.yml (with arm-64) installs

petsc                     3.20.2          real_hfe62963_100    conda-forge
petsc4py                  3.20.2          real_hd567e68_100    conda-forge
python                    3.12.3          h4a7b5fc_0_cpython    conda-forge

[guyer]

Fixed by #1064