Intermittent AIX jvmti.h build failure #2466

andrew-m-leonard · 2021-02-12T12:58:11Z

11:01:20  In file included from ./src/hotspot/share/interpreter/abstractInterpreter.cpp:45:
11:01:20  In file included from ./src/hotspot/share/prims/jvmtiExport.hpp:28:
11:01:20  ./build/aix-ppc64-server-release/hotspot/variant-server/gensrc/jvmtifiles/jvmti.h:1766:5: error: expected parameter declarator
11:01:20      );
11:01:20      ^
11:01:20  ./build/aix-ppc64-server-release/hotspot/variant-server/gensrc/jvmtifiles/jvmti.h:1770:5: error: expected parameter declarator
11:01:20      );
11:01:20      ^
11:01:20  ./build/aix-ppc64-server-release/hotspot/variant-server/gensrc/jvmtifiles/jvmti.h:1774:5: error: expected parameter declarator
11:01:20      );
11:01:20      ^
11:01:20  ./build/aix-ppc64-server-release/hotspot/variant-server/gensrc/jvmtifiles/jvmti.h:1778:5: error: expected parameter declarator
11:01:20      );
11:01:20      ^
11:01:20  ./build/aix-ppc64-server-release/hotspot/variant-server/gensrc/jvmtifiles/jvmti.h:1782:5: error: expected parameter declarator
11:01:20      );
11:01:20      ^
11:01:20  ./build/aix-ppc64-server-release/hotspot/variant-server/gensrc/jvmtifiles/jvmti.h:1786:5: error: expected parameter declarator
11:01:20      );
11:01:20      ^
11:01:20  ./build/aix-ppc64-server-release/hotspot/variant-server/gensrc/jvmtifiles/jvmti.h:1790:5: error: expected parameter declarator
11:01:20      );
11:01:20      ^
11:01:20  ./build/aix-ppc64-server-release/hotspot/variant-server/gensrc/jvmtifiles/jvmti.h:1794:5: error: expected parameter declarator
11:01:20      );
11:01:20      ^
11:01:20  ./build/aix-ppc64-server-release/hotspot/variant-server/gensrc/jvmtifiles/jvmti.h:1798:5: error: expected parameter declarator
11:01:20      );
11:01:20      ^
11:01:20  ./build/aix-ppc64-server-release/hotspot/variant-server/gensrc/jvmtifiles/jvmti.h:1802:5: error: expected parameter declarator
11:01:20      );
11:01:20      ^
11:01:20  ./build/aix-ppc64-server-release/hotspot/variant-server/gensrc/jvmtifiles/jvmti.h:1806:5: error: expected parameter declarator
11:01:20      );
11:01:20      ^
11:01:20  ./build/aix-ppc64-server-release/hotspot/variant-server/gensrc/jvmtifiles/jvmti.h:1810:5: error: expected parameter declarator
11:01:20      );
11:01:20      ^
11:01:20  ./build/aix-ppc64-server-release/hotspot/variant-server/gensrc/jvmtifiles/jvmti.h:1814:5: error: expected parameter declarator
11:01:20      );
11:01:20      ^
11:01:20  ./build/aix-ppc64-server-release/hotspot/variant-server/gensrc/jvmtifiles/jvmti.h:1824:38: error: use of undeclared identifier 'size'
11:01:20      return functions->Allocate(this, size, mem_ptr);
11:01:20                                       ^
11:01:20  ./build/aix-ppc64-server-release/hotspot/variant-server/gensrc/jvmtifiles/jvmti.h:1824:44: error: use of undeclared identifier 'mem_ptr'
11:01:20      return functions->Allocate(this, size, mem_ptr);
11:01:20                                             ^
11:01:20  ./build/aix-ppc64-server-release/hotspot/variant-server/gensrc/jvmtifiles/jvmti.h:1828:40: error: use of undeclared identifier 'mem'
11:01:20      return functions->Deallocate(this, mem);
11:01:20                                         ^
11:01:20  ./build/aix-ppc64-server-release/hotspot/variant-server/gensrc/jvmtifiles/jvmti.h:1832:44: error: 'thread' does not refer to a value
11:01:20      return functions->GetThreadState(this, thread, thread_state_ptr);
11:01:20                                             ^
11:01:20  /usr/include/sys/thread.h:105:8: note: declared here
11:01:20  struct  thread {
11:01:20          ^
11:01:20  In file included from ./src/hotspot/share/interpreter/abstractInterpreter.cpp:45:
11:01:20  In file included from ./src/hotspot/share/prims/jvmtiExport.hpp:28:
11:01:20  ./build/aix-ppc64-server-release/hotspot/variant-server/gensrc/jvmtifiles/jvmti.h:1832:52: error: use of undeclared identifier 'thread_state_ptr'
11:01:20      return functions->GetThreadState(this, thread, thread_state_ptr);
11:01:20                                                     ^
11:01:20  ./build/aix-ppc64-server-release/hotspot/variant-server/gensrc/jvmtifiles/jvmti.h:1836:46: error: use of undeclared identifier 'thread_ptr'
11:01:20      return functions->GetCurrentThread(this, thread_ptr);
11:01:20                                               ^
11:01:20  fatal error: too many errors emitted, stopping now [-ferror-limit=]
11:01:21  20 errors generated.
11:01:21  Error while processing ./src/hotspot/share/interpreter/abstractInterpreter.cpp.
11:01:21  gmake[3]: *** [/home/jenkins/workspace/build-scripts/jobs/jdk/jdk-aix-ppc64-hotspot/workspace/build/src/build/aix-ppc64-server-release/hlib/CompileJvm.gmk:141: recipe for target '/home/jenkins/workspace/build-scripts/jobs/jdk/jdk-aix-ppc64-hotspot/workspace/build/src/build/aix-ppc64-server-release/hotspot/variant-server/libjvm/objs/abstractInterpreter.o' failed
11:01:21  otspot/variant-server/libjvm/objs/abstractInterpreter.o] Error 1
11:01:21  gmake[3]: *** Waiting for unfinished jobs....
11:01:24  gmake[2]: *** [hotspot-server-libs] Error 1
11:01:24  gmake[2]: *** Waiting for unfinished jobs....
11:01:24  make/Main.gmk:252: recipe for target 'hotspot-server-libs' failed
11:02:25  
11:02:25  ERROR: Build failed for targets 'product-images legacy-jre-image test-image' in configuration 'aix-ppc64-server-release' (exit code 2) 
11:02:25  Stopping sjavac server
11:02:25  
11:02:25  === Output from failing command(s) repeated here ===
11:02:25  * For target hotspot_variant-server_libjvm_objs_abstractInterpreter.o:
11:02:25  In file included from ./src/hotspot/share/interpreter/abstractInterpreter.cpp:45:
11:02:25  In file included from ./src/hotspot/share/prims/jvmtiExport.hpp:28:
11:02:25  ./build/aix-ppc64-server-release/hotspot/variant-server/gensrc/jvmtifiles/jvmti.h:1766:5: error: expected parameter declarator
11:02:25      );
11:02:25      ^
11:02:25  ./build/aix-ppc64-server-release/hotspot/variant-server/gensrc/jvmtifiles/jvmti.h:1770:5: error: expected parameter declarator
11:02:25      );
11:02:25      ^
11:02:25  ./build/aix-ppc64-server-release/hotspot/variant-server/gensrc/jvmtifiles/jvmti.h:1774:5: error: expected parameter declarator
11:02:25      );
11:02:25      ^
11:02:25  ./build/aix-ppc64-server-release/hotspot/variant-server/gensrc/jvmtifiles/jvmti.h:1778:5: error: expected parameter declarator
11:02:25      );
11:02:25      ^
11:02:25  ./build/aix-ppc64-server-release/hotspot/variant-server/gensrc/jvmtifiles/jvmti.h:1782:5: error: expected parameter declarator
11:02:25     ... (rest of output omitted)
11:02:25  
11:02:25  * All command lines available in /home/jenkins/workspace/build-scripts/jobs/jdk/jdk-aix-ppc64-hotspot/workspace/build/src/build/aix-ppc64-server-release/make-support/failure-logs.
11:02:25  === End of repeated output ===

gensrc output:
jvmti.h.txt

The text was updated successfully, but these errors were encountered:

andrew-m-leonard · 2021-02-12T13:00:15Z

From GetAvailableProcessors onwards they are missing the 2nd argument...

  /*   144 : Get Available Processors */
  jvmtiError (JNICALL *GetAvailableProcessors) (jvmtiEnv* env,
    );

  /*   145 : Get Class Version Numbers */
  jvmtiError (JNICALL *GetClassVersionNumbers) (jvmtiEnv* env,
    );

andrew-m-leonard · 2021-02-12T13:14:19Z

Probably an upstream bug with src gen, perculiar to AIX it would seem...

andrew-m-leonard · 2021-02-15T11:20:09Z

Only seems to fail on node: build-osuosl-aix71-ppc64-1

andrew-m-leonard · 2021-02-16T07:16:30Z

Slightly different stack this time:
#2466

23:39:32  jvmtiGen error: java.lang.NullPointerException: Cannot load from object array because "this.m_extendedTypes" is nulljavax.xml.transform.TransformerException: java.lang.NullPointerException: Cannot load from object array because "this.m_extendedTypes" is null
23:39:32  	at java.xml/com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:797)
23:39:32  	at java.xml/com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:383)
23:39:32  	at jvmtiGen.main(jvmtiGen.java:137)
23:39:32  Caused by: java.lang.NullPointerException: Cannot load from object array because "this.m_extendedTypes" is null
23:39:32  	at java.xml/com.sun.org.apache.xml.internal.dtm.ref.sax2dtm.SAX2DTM2._type2(SAX2DTM2.java:1942)
23:39:32  	at java.xml/com.sun.org.apache.xml.internal.dtm.ref.sax2dtm.SAX2DTM2.getStringValueX(SAX2DTM2.java:2832)
23:39:32  	at java.xml/com.sun.org.apache.xml.internal.dtm.ref.sax2dtm.SAX2DTM2.getStringValue(SAX2DTM2.java:2930)
23:39:32  	at java.xml/com.sun.org.apache.xalan.internal.xsltc.dom.AdaptiveResultTreeImpl.getStringValue(AdaptiveResultTreeImpl.java:133)
23:39:32  	at jdk.translet/die.verwandlung.jvmti.template$dot$152()
23:39:32  	at jdk.translet/die.verwandlung.jvmti.applyTemplates18()
23:39:32  	at jdk.translet/die.verwandlung.jvmti.template$dot$107()
23:39:32  	at jdk.translet/die.verwandlung.jvmti.applyTemplates15()
23:39:32  	at jdk.translet/die.verwandlung.jvmti.template$dot$106()
23:39:32  	at jdk.translet/die.verwandlung.jvmti.applyTemplates15()
23:39:32  	at jdk.translet/die.verwandlung.jvmti.template$dot$102()
23:39:32  	at jdk.translet/die.verwandlung.jvmti.applyTemplates()
23:39:32  	at jdk.translet/die.verwandlung.jvmti.template$dot$100()
23:39:32  	at jdk.translet/die.verwandlung.jvmti.applyTemplates()
23:39:32  	at jdk.translet/die.verwandlung.jvmti.applyTemplates()
23:39:32  	at jdk.translet/die.verwandlung.jvmti.transform()
23:39:32  	at java.xml/com.sun.org.apache.xalan.internal.xsltc.runtime.AbstractTranslet.transform(AbstractTranslet.java:624)
23:39:32  	at java.xml/com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:790)
23:39:32  	... 2 more
23:39:32  ---------
23:39:32  java.lang.NullPointerException: Cannot load from object array because "this.m_extendedTypes" is null
23:39:32  	at java.xml/com.sun.org.apache.xml.internal.dtm.ref.sax2dtm.SAX2DTM2._type2(SAX2DTM2.java:1942)
23:39:32  	at java.xml/com.sun.org.apache.xml.internal.dtm.ref.sax2dtm.SAX2DTM2.getStringValueX(SAX2DTM2.java:2832)
23:39:32  	at java.xml/com.sun.org.apache.xml.internal.dtm.ref.sax2dtm.SAX2DTM2.getStringValue(SAX2DTM2.java:2930)
23:39:32  	at java.xml/com.sun.org.apache.xalan.internal.xsltc.dom.AdaptiveResultTreeImpl.getStringValue(AdaptiveResultTreeImpl.java:133)
23:39:32  	at jdk.translet/die.verwandlung.jvmti.template$dot$152()
23:39:32  	at jdk.translet/die.verwandlung.jvmti.applyTemplates18()
23:39:32  	at jdk.translet/die.verwandlung.jvmti.template$dot$107()
23:39:32  	at jdk.translet/die.verwandlung.jvmti.applyTemplates15()
23:39:32  	at jdk.translet/die.verwandlung.jvmti.template$dot$106()
23:39:32  	at jdk.translet/die.verwandlung.jvmti.applyTemplates15()
23:39:32  	at jdk.translet/die.verwandlung.jvmti.template$dot$102()
23:39:32  	at jdk.translet/die.verwandlung.jvmti.applyTemplates()
23:39:32  	at jdk.translet/die.verwandlung.jvmti.template$dot$100()
23:39:32  	at jdk.translet/die.verwandlung.jvmti.applyTemplates()
23:39:32  	at jdk.translet/die.verwandlung.jvmti.applyTemplates()
23:39:32  	at jdk.translet/die.verwandlung.jvmti.transform()
23:39:32  	at java.xml/com.sun.org.apache.xalan.internal.xsltc.runtime.AbstractTranslet.transform(AbstractTranslet.java:624)
23:39:32  	at java.xml/com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:790)
23:39:32  	at java.xml/com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:383)
23:39:32  	at jvmtiGen.main(jvmtiGen.java:137)
23:39:32  gmake[3]: *** [/home/jenkins/workspace/build-scripts/jobs/jdk16/jdk16-aix-ppc64-hotspot/workspace/build/src/build/aix-ppc64-server-release/hotspot/variant-server/gensrc/jvmtifiles/jvmti.html] Error 1
23:39:32  gmake[3]: *** Deleting file '/home/jenkins/workspace/build-scripts/jobs/jdk16/jdk16-aix-ppc64-hotspot/workspace/build/src/build/aix-ppc64-server-release/hotspot/variant-server/gensrc/jvmtifiles/jvmti.html'
23:39:32  gensrc/GensrcJvmti.gmk:106: recipe for target '/home/jenkins/workspace/build-scripts/jobs/jdk16/jdk16-aix-ppc64-hotspot/workspace/build/src/build/aix-ppc64-server-release/hotspot/variant-server/gensrc/jvmtifiles/jvmti.html' failed
23:39:32  gmake[3]: *** Waiting for unfinished jobs....
23:39:32  make/Main.gmk:244: recipe for target 'hotspot-server-gensrc' failed

aixtools · 2021-02-23T13:03:55Z

a) is there a 'grinder' string I can use to replicate this.
b) This says '11 days ago' - so I just want to mention that at Wed Feb 17 06:39:15 PST 2021 the AIXToolbox (aka yum update) was updated, but not the underlying OS.
c:) since 23 (today) - 11 (current age of issue) is 12 (before update) - perhaps whatever this was - was fixed by the OSS update.
d) back to a) - grinder to rinse and repeat - see if still valid.

andrew-m-leonard · 2021-02-23T13:09:38Z

@aixtools it fails during the build, so just kick off a build, doesn't fail every time

andrew-m-leonard · 2021-02-25T09:40:22Z

Going to do some frequency of failure tests, then git bisect back to beginning of year when we were not seeing this issue.

sxa · 2021-02-25T09:44:03Z

The two JDK17 builds I ran last night did not fail, although I have another three queued up on -1 (In addition to a couple of JDK8 ones which I'm using to do some performance testing

andrew-m-leonard · 2021-02-25T09:49:23Z

Test with --with-jobs=20 to see if it forces the issue: https://ci.adoptopenjdk.net/view/Failing%20Builds/job/build-scripts/job/jobs/job/jdk16/job/jdk16-aix-ppc64-hotspot/70/ Failed too big
try again with 12:
https://ci.adoptopenjdk.net/view/Failing%20Builds/job/build-scripts/job/jobs/job/jdk16/job/jdk16-aix-ppc64-hotspot/71/

andrew-m-leonard · 2021-02-25T11:08:51Z

Re-trying with jobs=1:
https://ci.adoptopenjdk.net/view/Failing%20Builds/job/build-scripts/job/jobs/job/jdk16/job/jdk16-aix-ppc64-hotspot/73/
=> Failed the same

andrew-m-leonard · 2021-02-25T12:41:13Z

Performing git bisect back to Dec1 2020

andrew-m-leonard · 2021-02-25T19:41:14Z

Searching backwards in jdk-17 for commit point that works:
1st attempt: 4thSep2020: https://github.com/AdoptOpenJDK/openjdk-jdk/commits/dev?before=76d6dea0694e0cdbdf873d59798ec063705db022+4001&branch=dev : https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk/job/jdk-aix-ppc64-hotspot/423/console => Failed
2nd attempt: Jun11th2020 : AdoptOpenJDK/openjdk-jdk@30ff2ad => Failed
3rd attempt: May8th2020 : 601891ac2b50111fee2e72bc6691f7e83a4bfb9b : https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk/job/jdk-aix-ppc64-hotspot/431/ => Failed
4th attempt: : Feb7th2020 : https://github.com/AdoptOpenJDK/openjdk-jdk/commits/dev?after=4b8d9abe2d20a8be4c1d93b0b45c809d5ea75311+7000&branch=dev : https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk/job/jdk-aix-ppc64-hotspot/435/ => Failed!

andrew-m-leonard · 2021-02-26T09:16:40Z

Trying a different tac, rebuilding 4th attempt again but with a jdk-14 bootjdk: Feb7th2020 : https://github.com/AdoptOpenJDK/openjdk-jdk/commits/dev?after=4b8d9abe2d20a8be4c1d93b0b45c809d5ea75311+7000&branch=dev : https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk/job/jdk-aix-ppc64-hotspot/440/console
Succeeds 5 times

andrew-m-leonard · 2021-02-26T09:28:33Z

Some themes:

Fails more easily on node build-osuosl-aix71-ppc64-1 than -2

andrew-m-leonard · 2021-02-26T14:31:01Z

The problem is due to the openj9 boot jdk being used, it seems jdk-15.0.2+7_openj9-0.24.0 when released in January and then got picked up as the boot jdk for building jdk16, causes the problem when running jvmtiGen on hotspot builds.
Simliarly jdk-16+36 openj9 build causes the same issue for jdk-17 builds.
Tests using a Hotspot bootjdk build fine:

jdk17 Hotspot AIX built with Hotspot bootjdk: https://ci.adoptopenjdk.net/view/work-in-progress/job/andrew-jdk-aix-ppc64-hotspot/
jdk17 OpenJ9 AIX built with Hotspot bootjdk: https://ci.adoptopenjdk.net/view/work-in-progress/job/andrew-jdk-aix-ppc64-openj9/
Raised issue: AIX jdk-15.0.2+7_openj9-0.24.0 or jdk-16 Nightly when used to build jdk16/17 Hotspot causes jvmtiGen failure eclipse-openj9/openj9#12061

sxa · 2021-02-28T12:28:47Z

So it was likely ok with openJ9 15.0.1? Do the OpenJ9 team know there's been a problem introduced with that level?

andrew-m-leonard · 2021-03-01T08:28:32Z

Yes issue raised with openj9: eclipse-openj9/openj9#12061

andrew-m-leonard added the buildbreak High priority issues that cause build breaks in jenkins or build scripts label Feb 12, 2021

M-Davies added the aix Issues that affect or relate to the AIX OS label Feb 12, 2021

M-Davies added the infra Issues that related to infrastructure problems or enhancements label Feb 15, 2021

sxa mentioned this issue Feb 24, 2021

build-osuosl-aix71-ppc64-1/2 missing libcurl.so ? adoptium/infrastructure#1946

Closed

andrew-m-leonard mentioned this issue Feb 25, 2021

Remove aix710 label from https://ci.adoptopenjdk.net/computer/test-ibm-aix71-ppc64-1/ until debugging finished adoptium/infrastructure#1978

Closed

This was referenced Feb 26, 2021

AIX jdk-15.0.2+7_openj9-0.24.0 or jdk-16 Nightly when used to build jdk16/17 Hotspot causes jvmtiGen failure eclipse-openj9/openj9#12061

Closed

Use AIX hotspot bootjdk to avoid issue #2466 #2498

Merged

M-Davies linked a pull request Feb 26, 2021 that will close this issue

Use AIX hotspot bootjdk to avoid issue #2466 #2498

Merged

M-Davies assigned andrew-m-leonard Feb 26, 2021

M-Davies added this to the February 2021 milestone Feb 26, 2021

karianna closed this as completed in #2498 Feb 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Intermittent AIX jvmti.h build failure #2466

Intermittent AIX jvmti.h build failure #2466

andrew-m-leonard commented Feb 12, 2021

andrew-m-leonard commented Feb 12, 2021

andrew-m-leonard commented Feb 12, 2021

andrew-m-leonard commented Feb 15, 2021

andrew-m-leonard commented Feb 16, 2021

aixtools commented Feb 23, 2021

andrew-m-leonard commented Feb 23, 2021

andrew-m-leonard commented Feb 25, 2021

sxa commented Feb 25, 2021

andrew-m-leonard commented Feb 25, 2021 •

edited

Loading

andrew-m-leonard commented Feb 25, 2021 •

edited

Loading

andrew-m-leonard commented Feb 25, 2021

andrew-m-leonard commented Feb 25, 2021 •

edited

Loading

andrew-m-leonard commented Feb 26, 2021 •

edited

Loading

andrew-m-leonard commented Feb 26, 2021

andrew-m-leonard commented Feb 26, 2021 •

edited

Loading

sxa commented Feb 28, 2021

andrew-m-leonard commented Mar 1, 2021

Intermittent AIX jvmti.h build failure #2466

Intermittent AIX jvmti.h build failure #2466

Comments

andrew-m-leonard commented Feb 12, 2021

andrew-m-leonard commented Feb 12, 2021

andrew-m-leonard commented Feb 12, 2021

andrew-m-leonard commented Feb 15, 2021

andrew-m-leonard commented Feb 16, 2021

aixtools commented Feb 23, 2021

andrew-m-leonard commented Feb 23, 2021

andrew-m-leonard commented Feb 25, 2021

sxa commented Feb 25, 2021

andrew-m-leonard commented Feb 25, 2021 • edited Loading

andrew-m-leonard commented Feb 25, 2021 • edited Loading

andrew-m-leonard commented Feb 25, 2021

andrew-m-leonard commented Feb 25, 2021 • edited Loading

andrew-m-leonard commented Feb 26, 2021 • edited Loading

andrew-m-leonard commented Feb 26, 2021

andrew-m-leonard commented Feb 26, 2021 • edited Loading

sxa commented Feb 28, 2021

andrew-m-leonard commented Mar 1, 2021

andrew-m-leonard commented Feb 25, 2021 •

edited

Loading

andrew-m-leonard commented Feb 25, 2021 •

edited

Loading

andrew-m-leonard commented Feb 25, 2021 •

edited

Loading

andrew-m-leonard commented Feb 26, 2021 •

edited

Loading

andrew-m-leonard commented Feb 26, 2021 •

edited

Loading