Semeru 21 hangs on macOS aarch64 when using the pkg JDK installation

[kgibm]

Semeru is installed using the pkg download of the JDK from https://developer.ibm.com/languages/java/semeru-runtimes/downloads/?version=21&os=macOS

$ java -version
openjdk version "21.0.4" 2024-07-16 LTS
IBM Semeru Runtime Open Edition 21.0.4.1 (build 21.0.4+7-LTS)
Eclipse OpenJ9 VM 21.0.4.1 (build openj9-0.46.1, JRE 21 Mac OS X aarch64-64-Bit 20240716_258 (JIT enabled, AOT enabled)
OpenJ9   - 4760d5d320
OMR      - 840a9adba
JCL      - db3fffb417c based on jdk-21.0.4+7)

Some executions of this java on the system path work fine but when trying to start OpenLiberty, Semeru hangs indefinitely; however, if I add the JDK to PATH explicitly using export PATH=/Library/Java/JavaVirtualMachines/ibm-semeru-open-21.jdk/Contents/Home/bin/:${PATH}, then it works fine, so this issue seems to be something about the Semeru pkg installation of /usr/bin/java in the system.

Reproduction steps:

1. Download and double click pkg version of the JDK: https://developer.ibm.com/languages/java/semeru-runtimes/downloads/?version=21&os=macOS
2. Open Terminal and ensure that java points to /usr/bin/java:

   $ which java
   /usr/bin/java
   

3. Download latest OpenLiberty release:

   wget https://public.dhe.ibm.com/ibmdl/export/pub/software/openliberty/runtime/release/24.0.0.11/openliberty-24.0.0.11.zip
   

4. Unzip:

   unzip openliberty*zip
   

5. Create OpenLiberty server (note that this is a Java execution which works fine):

   $ wlp/bin/server create server1
   
   Server server1 created.
   

6. Start OpenLiberty Server:

   $ wlp/bin/server start server1
   
   
   

7. Both Java processes are hung indefinitely. Open a new Terminal tab to find the PIDs (the one with --status:start is the Java process trying to start the actual OpenLiberty Java server process):

   $ ps auxwww | grep java
   kevin            35867   0.0  0.0 410059776     48 s007  S+   10:47AM   0:38.74 /usr/bin/java -javaagent:/tmp/play/wlp/bin/tools/ws-javaagent.jar -Djava.awt.headless=true -Djdk.attach.allowAttachSelf=true -jar /tmp/play/wlp/bin/tools/ws-server.jar server1
   kevin            35868   0.0  0.0 410597088   2240 s007  R+   10:47AM   0:38.73 /usr/bin/java -javaagent:/tmp/play/wlp/bin/tools/ws-javaagent.jar -Djava.awt.headless=true -Djdk.attach.allowAttachSelf=true -jar /tmp/play/wlp/bin/tools/ws-server.jar server1 --pid=35867 --status:start
   kevin            36030   0.0  0.0 410724096   1360 s008  S+   10:48AM   0:00.00 grep java
   

8. First, send kill -3 to the OpenLiberty server process (without --status:start):

   kill -3 35867
   

9. The current working directory of that process should be wlp/usr/servers/server1 but there's no javacore:

   $ ls wlp/usr/servers/server1/javacore*
   ls: wlp/usr/servers/server1/javacore*: No such file or directory
   

10. Send kill -3 to the other process and it core dumps in the other Terminal tab:

    $ wlp/bin/server start server1
    
    wlp/bin/server: line 887: 35868 Quit: 3                 (core dumped) "${JAVA_CMD}" "$@"
    

11. Load core dump in lldb and there's only one thread and here's the stack:

    # lldb -c /cores/core.35868
    (lldb) target create --core "/cores/core.35868"
    Core file '/cores/core.35868' (arm64) was loaded.
    (lldb) thread list
    Process 0 stopped
    * thread #1: tid = 0x0000, 0x000000018de5f840 dyld`close + 8
    (lldb) bt
    * thread #1
      * frame #0: 0x000000018de5f840 dyld`close + 8
        frame #1: 0x000000018dec547c dyld`closefd + 48
        frame #2: 0x000000018dec6358 dyld`jettison + 188
        frame #3: 0x000000018de67820 dyld`dyld4::CacheFinder::~CacheFinder() + 48
        frame #4: 0x000000018de67c80 dyld`dyld4::ProcessConfig::DyldCache::DyldCache(dyld4::ProcessConfig::Process&, dyld4::ProcessConfig::Security const&, dyld4::ProcessConfig::Logging const&, dyld4::SyscallDelegate&, lsl::Allocator&) + 1076
        frame #5: 0x000000018de65898 dyld`dyld4::ProcessConfig::ProcessConfig(dyld4::KernelArgs const*, dyld4::SyscallDelegate&, lsl::Allocator&) + 128
        frame #6: 0x000000018de62dc0 dyld`start + 1560
    

12. Adding the Semeru installation to PATH fixes the issue:

    $ export PATH=/Library/Java/JavaVirtualMachines/ibm-semeru-open-21.jdk/Contents/Home/bin/:${PATH}
    $ java -version
    openjdk version "21.0.4" 2024-07-16 LTS
    IBM Semeru Runtime Open Edition 21.0.4.1 (build 21.0.4+7-LTS)
    Eclipse OpenJ9 VM 21.0.4.1 (build openj9-0.46.1, JRE 21 Mac OS X aarch64-64-Bit 20240716_258 (JIT enabled, AOT enabled)
    OpenJ9   - 4760d5d320
    OMR      - 840a9adba
    JCL      - db3fffb417c based on jdk-21.0.4+7)
    $ bin/server start server1
    
    Starting server server1.
    Server server1 started with process ID 37560.
    

13. kill -3 works as expected:

    $ kill -3 37560
    $ ls usr/servers/server1/javacore*
    usr/servers/server1/javacore.20241118.122039.37560.0001.txt
    

macOS version:

$ system_profiler SPSoftwareDataType SPHardwareDataType | grep -v -e UUID -e UDID -e 'User Name' -e 'Computer Name' -e Serial
Software:

    System Software Overview:

      System Version: macOS 14.6.1 (23G93)
      Kernel Version: Darwin 23.6.0
      Boot Volume: Macintosh HD
      Boot Mode: Normal
      Secure Virtual Memory: Enabled
      System Integrity Protection: Disabled
      Time since boot: 2 hours, 46 minutes

Hardware:

    Hardware Overview:

      Model Name: MacBook Pro
      Model Identifier: Mac14,10
      Model Number: Z1740017JLL/A
      Chip: Apple M2 Pro
      Total Number of Cores: 12 (8 performance and 4 efficiency)
      Memory: 32 GB
      System Firmware Version: 10151.140.19
      OS Loader Version: 10151.140.19
      Activation Lock Status: Disabled