DaaLoadTest_daa -Xgcpolicy:balanced Unable to locate JIT stack map

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk8_j9_special.system_aarch64_linux_Personal_testList_0/148 - ubu22-aarch64-2
DaaLoadTest_daa1_special_5m_16
-XX:+UseCompressedOops -Xgcpolicy:balanced -Xjit:counts=- - - - - - 1 1 1 1000 250 250 - - - 10000 100000 10000,gcOnResolve,rtResolve,sampleInterval=2,scorchingSampleThreshold=10000,quickProfile -Xcheck:gc:vmthreads:all:quiet

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk8_j9_special.system_aarch64_linux_Personal_testList_0/148/system_test_output.tar.gz

12:19:25  DLT stderr JVMCDRT000E Unable to locate JIT stack map - aborting VM
12:19:25  DLT stderr JVMCDRT001E Method: com/ibm/dataaccess/DecimalData.slowSignedPackedToBigDecimal([BIIIZ)Ljava/math/BigDecimal; (0000000000484BF0)
12:19:25  DLT stderr JVMCDRT002E Failing PC: 0000FFFFA0598609 (offset 0000000000000081), metaData = 0000FFFF3FC41F08
12:19:25  DLT stderr 17:19:24.535 0x457600j9codertvm(j9ji.110    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDK8_aarch64_linux_Personal/openj9/runtime/codert_vm/jswalk.c:534: ((0 ))

DaaLoadTest_daa2_special_5m_16

12:46:40  DLT stderr JVMCDRT000E Unable to locate JIT stack map - aborting VM
12:46:40  DLT stderr JVMCDRT001E Method: com/ibm/dataaccess/DecimalData.slowSignedPackedToBigDecimal([BIIIZ)Ljava/math/BigDecimal; (00000000004862B0)
12:46:40  DLT stderr JVMCDRT002E Failing PC: 0000FFFF8FEAB709 (offset 0000000000000089), metaData = 0000FFFF8C4C4108
12:46:40  DLT stderr 17:46:38.935 0x319a00j9codertvm(j9ji.110    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDK8_aarch64_linux_Personal/openj9/runtime/codert_vm/jswalk.c:534: ((0 ))

https://openj9-jenkins.osuosl.org/job/Test_openjdk8_j9_special.system_aarch64_linux_Personal_testList_1/149/
DaaLoadTest_all_special_5m_13 -XX:+UseCompressedOops -Xjit -Xgcpolicy:balanced

10:36:44  DLT stderr JVMCDRT000E Unable to locate JIT stack map - aborting VM
10:36:44  DLT stderr JVMCDRT001E Method: com/ibm/dataaccess/DecimalData.convertPackedDecimalToBigDecimal([BIIIZ)Ljava/math/BigDecimal; (0000000000484A10)
10:36:44  DLT stderr JVMCDRT002E Failing PC: 0000F63C1AB5C685 (offset 000000000000006D), metaData = 0000F63C19E19988
10:36:44  DLT stderr 15:36:43.550 0xf1800j9codertvm(j9ji.110    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDK8_aarch64_linux_Personal/openj9/runtime/codert_vm/jswalk.c:534: ((0 ))

https://openj9-jenkins.osuosl.org/job/Test_openjdk8_j9_special.system_aarch64_linux_Personal_testList_3/148/
DaaLoadTest_daa2_special_5m_12 -Xjit -Xgcpolicy:balanced -Xnocompressedrefs
DaaLoadTest_all_special_5m_16
-XX:+UseCompressedOops -Xgcpolicy:balanced -Xjit:counts=- - - - - - 1 1 1 1000 250 250 - - - 10000 100000 10000,gcOnResolve,rtResolve,sampleInterval=2,scorchingSampleThreshold=10000,quickProfile -Xcheck:gc:vmthreads:all:quiet

https://openj9-jenkins.osuosl.org/job/Test_openjdk8_j9_special.system_aarch64_linux_Personal_testList_4/148/
DaaLoadTest_daa2_special_5m_13

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_special.system_aarch64_linux_Personal/192/tapTestReport/

17 failed to build the JVM

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_special.system_aarch64_linux_Personal/85/tapTestReport/

https://openj9-jenkins.osuosl.org/job/Test_openjdk24_j9_special.system_aarch64_linux_Personal/5/tapTestReport/

Changes from the last weekly build
c9b8510...d69c8c9
eclipse-openj9/openj9-omr@9953428...c6b19ea

[pshipton]

@hzongaro @knn-k pls take a look.

[knn-k]

Some test modes for balanced GC on AArch64 were turned on by adoptium/TKG#530 last week.
I ran extended.system tests before that, but I didn't try special.system.

I opened adoptium/TKG#676 as a workaround for the time being.
I will investigate the test failures.

[knn-k]

I can recreate some of the test failures using Java 11 locally. The tests run successfully by excluding DecimalData methods from JIT compilation using the following environment variable.

export OPENJ9_JAVA_OPTIONS="-Xjit:exclude={com/ibm/dataaccess/DecimalData.*}  -XX:+MergeCompilerOptions"

I can recreate the test failure by running DaaLoadTest_daa2_special_5m_12 (Mode 501) by limiting the Java method to compile with:

export OPENJ9_JAVA_OPTIONS="-Xjit:limit={com/ibm/dataaccess/DecimalData.slowSignedPackedToBigDecimal(*} -XX:+MergeCompilerOptions"

The class DecimalData is used by net.openj9.test.decimals.DecimalTestBase.

[knn-k]

Interestingly, it seems I cannot recreate the test failure on AArch64 macOS.
I ran DaaLoadTest_daa2_special_5m_12 (Mode 501) with Java 11 macOS 20 times (10x in local environment + 10x in Grinder), and the test passes.
https://openj9-jenkins.osuosl.org/job/Grinder/4132

[knn-k]

I am trying to get to the bottom of the failure.
I ran the DaaLoadTest_daa2_special_5m_12 and DaaLoadTest_daa2_special_5m_16 tests by disabling inlining of newarray, and the tests failed. I need to look into other parts of the JITed code.

[knn-k]

I disabled some test modes with DaaLoadTest_daa1_special_5m, DaaLoadTest_daa2_special_5m, and DaaLoadTest_all_special_5m by adoptium/aqa-tests#5989 for the time being.

On the other hand, PR #20821 was merged today. It enables the off-heap allocation as default for balanced GC.

[knn-k]

I tried running DaaLoadTest_*_special_5m tests using Java 21 on AArch64 Linux with the off-heap allocation enabled in the following jobs:

• https://openj9-jenkins.osuosl.org/job/Grinder/4138
• https://openj9-jenkins.osuosl.org/job/Grinder/4139

It is much stabler than off-heap allocation being disabled.
DaaLoadTest_*_special_5m_13 (Mode551) tests fail in those jobs during the JIT optimization phase (global value propagation) as shown below:

[2025-02-28T03:10:24.211Z] DLT stderr Method_being_compiled=java/math/BigInteger.smallToString(ILjava/lang/StringBuilder;I)V
[2025-02-28T03:10:24.211Z] DLT stderr Target=2_90_20250227_565 (Linux 5.15.0-133-generic)
[2025-02-28T03:10:24.211Z] DLT stderr CPU=aarch64 (4 logical CPUs) (0x1ef88d000 RAM)
[2025-02-28T03:10:24.211Z] DLT stderr ----------- Stack Backtrace -----------
[2025-02-28T03:10:26.119Z] DLT stderr _ZN2TR19VPMergedConstraints12intIntersectEPNS_12VPConstraintEP11ListElementIS1_EPN3OMR16ValuePropagationE+0xc4 (0x0000FFFF827D0554 [libj9jit29.so+0x820554])
[2025-02-28T03:10:26.119Z] DLT stderr _ZN2TR19VPMergedConstraints10intersect1EPNS_12VPConstraintEPN3OMR16ValuePropagationE+0xec (0x0000FFFF827D574C [libj9jit29.so+0x82574c])
[2025-02-28T03:10:26.119Z] DLT stderr _ZN2TR12VPConstraint9intersectEPS0_PN3OMR16ValuePropagationE+0x44 (0x0000FFFF827CF204 [libj9jit29.so+0x81f204])
[2025-02-28T03:10:26.119Z] DLT stderr _ZN3OMR16ValuePropagation19addConstraintToListEPN2TR4NodeEiiPNS1_12VPConstraintEP12TR_HedgeTreeINS0_15ValueConstraintEEb+0x754 (0x0000FFFF8280BEC8 [libj9jit29.so+0x85bec8])

Mode551 uses the options: -XX:+UseCompressedOops -Xjit -Xgcpolicy:balanced

[knn-k]

The failure with DaaLoadTest_*_special_5m_13 above seems to be platform-independent. I can reproduce it on x86 Linux. See https://openj9-jenkins.osuosl.org/job/Grinder/4141 (5/5 failure)

I opened a separate issue. #21229