Define policy for compiler upgrades on our machines

[sxa]

At present we have an ad-hoc mechanism for determining when to perform compiler upgrades. the process is now reasonably well defined (although documentation needs to be collated) but we should also have a policy of checking for critical security upgrades and determining when to perform upgrades for upcoming OpenJDK releases and/or updating the compilers for old machines and verifying that they get deployed in a timely manner.

[sxa]

To anyone picking this up - it might be useful to understand what other providers are doing in terms of compiler versions for each platform.

[jerboaa]

It would be great if there was a trail of documentation. Which JDK uses which compiler/os version etc.

[sxa]

It would be great if there was a trail of documentation. Which JDK uses which compiler/os version etc.

Yeah totally agree (although I'll note that convincing people to keep it up to date is a continual challenge). At the moment that's all in the https://github.com/adoptium/temurin-build/tree/master/build-farm/platform-specific-configurations scripts but making that more visible would be a very useful thing to do alongside this.

(I'll also add the link to the upstream supported environments since I didn't put that in the initial description here - thanks for the reminder today @jerboaa)

[sxa]

See also:

• Should we use GCC11 for building Alpine JDK19+? temurin-build#3001
• Smoke test to check that correct compiler was used temurin-build#2826
• Investigate changing jdk8u to build with VS2019 for Windows for July 2022 PSU temurin-build#2922
• PPC building of openjdk could be better optimized for pwr5+ temurin-build#2560
• Align JDK 16 compiler toolchains with upstream temurin-build#2432
• Ensure all Windows build machines have same compiler level #1270

[sxa]

@andrew-m-leonard Would be good to have your input on this, especially in light of the number of outstanding issues mentioned in the previous comment. I know we also spoke about getting JDK <20 moved to building on the Windows Server 2022 machines - I don't think I've seen that change so is that still feasible or are we too late to make that happen now for the April cycle?

[andrew-m-leonard]

@sxa I think with only 1 week to go, i'd rather stabilize on what we currently have, unless there's a particular reason to move for this release?

[sxa]

@sxa I think with only 1 week to go, i'd rather stabilize on what we currently have, unless there's a particular reason to move for this release?

OK but we do need to try and get some attention on this post-release - most of the issues above haven't seen much action and we need to ensure we remain as current as practical and consistent with what the upstream project supports.

[sxa]

All fairly standard stuff so fundamentally no objections to using that as a policy, although we still need to decide the details of 3.2 in terms of when we choose to update (outside critical security updates)

For example, in the latest docs it says "The minimum accepted version of gcc is 5.0. Older versions will generate a warning by configure and are unlikely to work. The JDK is currently known to be able to compile with at least version 11.2 of gcc. In general, any version between these two should be usable."

Should we aim to be tracking Oracle's (which is the 11.2 that it's "known" to compile with)?

I'll note that we should typically be able to handle 3.2 by regular ansible updates putting the newer versions on, but that's not the case at present. And I'll also note that all of this is applicable to the levels of boot JDK which we have installed on the machines.

We could add reproducibility testing to 4.2 as well (It is unlikely to be reproducible pre and post upgrade, but verifying that it's the same for two builds of the updated compiler would make sense.

Initially for 4.4 we can probably cover that with a relevant GitHub issue for the update.

[CarmenDelgadoEclipse]

FYI @fredg02 :)

[sxa]

Only comment is that I don't know if we need 3.2 and 3.3 to be separate - 3.2 has "Schedule and perform compiler upgrades" and that should always be applied to all existing or future machines to ensure they are consistent.

Otherwise I'm happy with this.

[gdams]

Agreed! how about this @sxa:

Policy for Compiler Upgrades on Adoptium Build Machines

1. Objective

The objective of this policy is to establish a systematic and efficient approach to upgrading compilers on Adoptium build machines. This policy aims to ensure timely updates, reduce security risks, maintain compatibility with OpenJDK releases, and enhance overall system performance.

2. Scope

This policy applies to all compilers utilised on Adoptium build machines, including any other tools or utilities that contribute to the build process.

3. Compiler Upgrade Policy

3.1. Security Updates

The infrastructure/security team will continuously monitor relevant sources, for critical security updates. In the case of any critical security update, the team will:

• Assess the potential impact on Adoptium build machines.
• Prioritise the update and develop a timeline for deployment.
• Notify the relevant parties and coordinate with them to upgrade the compilers with minimal disruption.

3.2. OpenJDK Releases

To maintain compatibility with OpenJDK releases and ensure consistency across all machines, the team will:

• Track the OpenJDK release cycle and compiler version requirements, aiming to follow Oracle's version of gcc (11.2).
• Test compatibility with new compiler versions in a staging environment before deployment.
• Schedule and perform compiler upgrades in sync with OpenJDK releases, ensuring minimal impact on ongoing projects.
• Review compiler levels each time a new major release of OpenJDK is added to the CI system.
• Regularly evaluate the levels of boot JDK installed on the machines and prioritize their upgrades accordingly.

Additionally, for older build machines, the team will:

• Regularly evaluate compiler performance and compatibility with current build requirements.
• Identify any build machines that require compiler updates and prioritize them accordingly.
• Develop an upgrade plan and coordinate with the relevant parties for deployment.

4. Upgrade Process

4.1. Notification

The team will notify all relevant stakeholders about the planned compiler upgrade, including the expected timeframe, potential impact, and any required actions on their part.

4.2. Testing

Before deploying compiler updates on production machines, the team will:

• Test the updates in a controlled staging environment to ensure compatibility, performance, and reproducibility.
• Identify and resolve any issues or conflicts that may arise during the upgrade process.

4.3. Deployment

The infrastructure team will coordinate the deployment of compiler upgrades on the build machines, ensuring minimal disruption to Adoptium projects.

4.4. Documentation

The team will maintain detailed documentation for each upgrade, including the reason for the upgrade, the new compiler version, and any issues encountered and resolved during the process. This documentation will be readily accessible for future reference and audits, and can be recorded in relevant GitHub issues.

5. Monitoring and Review

The team will continuously monitor the performance and stability of the updated compilers on the build machines. They will also review this policy annually or as needed to ensure its effectiveness and relevance.

[sxa]

In the absence of further comments I'm going to consider this closed and it can be revised as and when required.