Unexpected tar error while un tarring jdk17 binary in ppc64le and arm32 ubuntu 2404 docker image #3547

Haroon-Khel · 2024-05-02T17:17:43Z

Hitting a tar error while building arm32 and ppc64le ubuntu 24.04 docker static containers

 > [ 7/25] RUN mkdir -p /usr/lib/jvm/jdk17 && tar -xpzf /tmp/jdk17.tar.gz -C /usr/lib/jvm/jdk17 --strip-components=1:                                                                                               
0.295 tar: conf/security/policy/unlimited: Cannot change mode to rwxr-xr-x: Operation not permitted                                                                                                                 
0.295 tar: conf/security/policy/limited: Cannot change mode to rwxr-xr-x: Operation not permitted                                                                                                                   
0.295 tar: conf/security/policy: Cannot change mode to rwxr-xr-x: Operation not permitted                                                                                                                           
0.295 tar: conf/security: Cannot change mode to rwxr-xr-x: Operation not permitted                                                                                                                                  
0.295 tar: conf/sdp: Cannot change mode to rwxr-xr-x: Operation not permitted
0.296 tar: conf/management: Cannot change mode to rwxr-xr-x: Operation not permitted
0.296 tar: conf: Cannot change mode to rwxr-xr-x: Operation not permitted
0.305 tar: legal/java.base: Cannot change mode to rwxr-xr-x: Operation not permitted
1.052 tar: jmods: Cannot change mode to rwxr-xr-x: Operation not permitted

The text was updated successfully, but these errors were encountered:

Haroon-Khel · 2024-05-02T17:21:21Z

The binaries untar without error on my local machine

sxa · 2024-05-03T10:13:06Z

Interesting ... yeah I can replicate that on one of my arm32 systems.
That's really odd ...
It's not specific to our tar file, but seems to be affecting directories extracted by tar. Sounds like a bug in the new ubuntu unless it's related to the kernel on the host. Like you I couldn't replicate with an aarch64 container with either Ubuntu 20.04 or 22.04 as the host machine. tar is at the latest version.
As an interim measure I would propose doing chmod -R a+rX /usr/lib/jvm/jdk-17-* afterwards which seems to work without problems but I'm nervous about whether this means we'll see issues elsewhere in our testing ...

sxa · 2024-05-07T09:15:38Z

Thought I'd already added this comment (Edit: yes I did but at #3501 (comment)) but running an emulated ppc64le container on another 24.04 host system did not show a problem, which works suggest there isn't a fundamentally problem with the base container and it potentially is related to the kernel being used

sxa · 2024-05-07T09:16:34Z

This is a situation where having QPC updated with latest images would help.

adamfarley · 2024-07-08T13:51:21Z

JDK11 Special.openjdk, Extended.system, and Special.functional all have appear to have a similar-looking issue unpacking the build with a tar command:

22:43:12 Uncompressing file: OpenJDK11U-jdk_ppc64le_linux_hotspot_11.0.24_7-ea.tar.gz ...
22:43:14 tar: jdk-11.0.24+7/man/ja: Cannot change mode to rwxrwxr-x: Operation not permitted
22:43:20 tar: jdk-11.0.24+7/legal/jdk.jartool/LICENSE: Cannot change mode to rwxrwxr-x: Operation not permitted
22:43:20 tar: jdk-11.0.24+7/legal/jdk.jartool/ADDITIONAL_LICENSE_INFO: Cannot change mode to rwxrwxr-x: Operation not permitted
22:43:20 tar: jdk-11.0.24+7/legal/jdk.jartool/ASSEMBLY_EXCEPTION: Cannot change mode to rwxrwxr-x: Operation not permitted
22:43:20 tar: jdk-11.0.24+7/legal/jdk.internal.jvmstat/LICENSE: Cannot change mode to rwxrwxr-x: Operation not permitted
22:43:20 tar: jdk-11.0.24+7/legal/jdk.internal.jvmstat/ADDITIONAL_LICENSE_INFO: Cannot change mode to rwxrwxr-x: Operation not permitted
22:43:20 tar: jdk-11.0.24+7/legal/jdk.internal.jvmstat/ASSEMBLY_EXCEPTION: Cannot change mode to rwxrwxr-x: Operation not permitted
etc etc

All three happen on a 24.04 docker host.

Haroon-Khel · 2024-07-31T14:32:40Z

At the moment the 2 problem machines are test-docker-ubuntu2404-armv7-2 and test-docker-ubuntu2404-ppc64le-1. Another ubuntu2404 arm32 node is test-docker-ubuntu2404-armv7-1 and this problem does not occur on this machine. Same tar versions

jenkins@299a170b9f8f:~$ tar --version
tar (GNU tar) 1.35
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.

adamfarley · 2024-11-25T15:19:30Z

This happened again on test-docker-ubuntu2404-armv7-2 on 2024/11/20
https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk17u/job/jdk17u-linux-arm-temurin_SmokeTests/259/console

sxa · 2024-11-25T18:33:33Z

Interesting - that's an Ubuntu 24 container on an Ubuntu 22 host.

sxa · 2024-11-25T18:58:04Z

I can replicate this. All of the offending files appear to be symbolic links to files in legal/java.base. It only causes a problem with tar - Running chmod on the files afterwards is ok.

jenkins@dockerhost-equinix-ubuntu2204-armv8-1:~$ docker run -it aqa_u2404_arm32 bash
root@33aee01c5a84:/# wget https://github.com/adoptium/temurin17-binaries/releases/download/jdk-17.0.13%2B11/OpenJDK17U-jdk_arm_linux_hotspot_17.0.13_11.tar.gz
root@33aee01c5a84:/# tar xfz OpenJDK17U-jdk_arm_linux_hotspot_17.0.13_11.tar.gz

agent	host	result
test-docker-ubuntu2004-armv7l-3	dockerhost-equinix-ubuntu2404-armv8-1	✅ link
test-docker-ubuntu2404-armv7-2	dockerhost-equinix-ubuntu2204-armv8-1	❌ link
test-docker-ubuntu2004-armv7l-6	dockerhost-equinix-ubuntu2204-armv8-1	✅ link

https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk17u/job/jdk17u-linux-arm-temurin_SmokeTests/265/

Based on the above this is almost certainly due to running a later version of Ubuntu (which has glibc/kernel interdependencies that are too new) on an older kernel. The other possibility (which I shall aim to check tomorrow) is whether restarting docker resolves it (potentially if it has been upgraded) and also whether there are any pending docker package updates on the host that we might apply which might affect this.

sxa · 2024-11-26T11:09:09Z

Restarting docker made no difference. Restarting the machine made no difference.
An aarch64 Ubuntu 24.04 container on the host works ok, so this is specific to arm32 Ubuntu 24.04 containers on Ubuntu 22.04 host.

Solution here is to deactivate test-docker-ubuntu2404-armv7l-2 which I've done.

sxa · 2024-11-29T14:13:29Z

FYI @Haroon-Khel we probably want to just decommission this particular machine now.

sxa · 2024-12-16T11:04:19Z

@Haroon-Khel If you're happy with the analysis above can you add this to your list for this iteration please?

Haroon-Khel · 2025-02-10T19:41:07Z

Did a bit more digging on this. Looks like its a bug in ubuntu2404 or docker which prevents the container from running the fchmodat2 system call
docker/docker-ce-packaging#1007 (comment)

If I run the ubuntu 2404 docker container with --security-opt seccomp=unconfined, I can untar files without error. I believe this option allows the container to unrestricted system calls so I am not sure this is good security wise.

ocaml/infrastructure#121 (comment) suggests Docker >= 25.0.3 and libseccomp2 >= 2.5.5 solves this, so its a matter of upgrading those packages on the problem dockerhosts

root@dockerhost-osuosl-ubuntu2404-ppc64le-1:~# docker version
Client:
 Version:           26.1.3
 API version:       1.43 (downgraded from 1.45)
 Go version:        go1.22.2
 Git commit:        26.1.3-0ubuntu1~24.04.1
 Built:             Mon Oct 14 14:29:26 2024
 OS/Arch:           linux/ppc64le
 Context:           default

Server:
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.22.2
  Git commit:       24.0.7-0ubuntu4.1
  Built:            Fri Aug  9 02:33:20 2024
  OS/Arch:          linux/ppc64le
  Experimental:     false
 containerd:
  Version:          1.7.24
  GitCommit:        
 runc:
  Version:          1.1.12-0ubuntu3.1
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:

Haroon-Khel · 2025-02-10T19:52:14Z

To do:

Upgrade docker on both ppc64le dockerhosts to >= 25.0.3
Upgrade libseccomp2 on the ubuntu 2204 arm64 dockerhost and ubuntu 2004 ppc64le dockerhost to >= 2.5.5
- I suspect ubuntu 2204 and 2004 wont allow this so may have to upgrade the OS to ubuntu 2404

sxa · 2025-02-10T20:02:07Z

* Upgrade libseccomp2 on the ubuntu 2204 arm64 dockerhost to >= 2.5.5
  * I suspect ubuntu 2204 wont allow this so may have to upgrade the OS to ubuntu 2404

I would test with the latest available if it doesn't have 2.5.5 in the repositories. Ubuntu (and other LTS distribution providers) will often backport important patches so even if they're showing something earlier than 2.5.5 it may be ok.

Haroon-Khel · 2025-02-11T11:21:51Z

Upgraded docker on dockerhost-osuosl-ubuntu2404-ppc64le-1 to v27

Server: Docker Engine - Community
 Engine:
  Version:          v27.4.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.10
  Git commit:       c710b88
  Built:            Mon Dec 23 11:56:44 2024
  OS/Arch:          linux/ppc64le
  Experimental:     false

I was able to untar a jdk binary on a ubuntu 2404 container on it without the permissions error. Looks good

Haroon-Khel · 2025-02-11T12:30:22Z

On the arm64 dockerhosts, it looks like the tar error on arm32 ubuntu 2404 containers cleared itself up? Docker or libseccomp2 may have upgraded during an automated patch. I cant seem to recreate the tar error on a arm32 ubuntu2404 container on any of the arm64 docker nodes

Haroon-Khel · 2025-02-11T15:05:41Z

dockerhost-skytap-ubuntu2004-ppc64le-1 is the remaining problem machine. As per #3588 I am going to upgrade it to Ubuntu 2404, so I will upgrade docker on the machine after the OS upgrade (if ill still need to)

Haroon-Khel · 2025-02-17T16:53:47Z

I am starting the dockerhost-skytap-ubuntu2004-ppc64le-1 OS upgrade right now

Haroon-Khel · 2025-02-17T17:24:49Z

The upgrade terminated midway due to a lack of diskspace, presumably on /

Filesystem                      Size  Used Avail Use% Mounted on
/dev/mapper/ubu1604p8--vg-root   38G   28G  8.2G  77% /

Haroon-Khel · 2025-02-18T14:18:30Z

Reiterating #3588 (comment)

Ive upgraded dockerhost-skytap-ubuntu2004-ppc64le-1 to ubuntu 22.04, disk space issues on /boot are preventing an upgrade to ubuntu 24.04 but I am comfortable keeping it on 22 since its still supported. Its ubuntu2404 container, which has long been offline due to the untarring issue, is now able to run grinders because Docker on the host system has been upgraded to v26
https://ci.adoptium.net/job/Grinder/12705/console

github-actions bot added arch:arm32 arch:ppc64le (pLinux) docker labels May 2, 2024

adamfarley mentioned this issue Jul 8, 2024

Build Issue Summary for 2024-07-08 adoptium/temurin-build#3878

Closed

smlambert added this to Adoptium Backlog Sep 5, 2024

github-project-automation bot moved this to Todo in Adoptium Backlog Sep 5, 2024

adamfarley mentioned this issue Nov 25, 2024

Build Issue Summary for 2024-11-25 adoptium/temurin-build#4056

Closed

sxa added this to 2024 4Q Adoptium Plan Nov 25, 2024

sxa removed this from Adoptium Backlog Nov 25, 2024

sxa self-assigned this Nov 25, 2024

adamfarley mentioned this issue Dec 11, 2024

Build Issue Summary for 2024-12-09 adoptium/temurin-build#4081

Closed

sxa assigned Haroon-Khel Dec 20, 2024

Haroon-Khel mentioned this issue Feb 10, 2025

EPIC: Upgrade EOL Ubuntu Machines To New 24.04 LTS Version #3588

Closed

8 tasks

Haroon-Khel closed this as completed Feb 18, 2025

github-project-automation bot moved this to Done in 2024 4Q Adoptium Plan Feb 18, 2025

Haroon-Khel added this to 2025 Adoptium Plan Feb 18, 2025

Haroon-Khel removed this from 2024 4Q Adoptium Plan Feb 18, 2025

Haroon-Khel moved this to Done in 2025 Adoptium Plan Feb 18, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unexpected tar error while un tarring jdk17 binary in ppc64le and arm32 ubuntu 2404 docker image #3547

Unexpected tar error while un tarring jdk17 binary in ppc64le and arm32 ubuntu 2404 docker image #3547

Haroon-Khel commented May 2, 2024

Haroon-Khel commented May 2, 2024

sxa commented May 3, 2024

sxa commented May 7, 2024 •

edited

Loading

sxa commented May 7, 2024 •

edited

Loading

adamfarley commented Jul 8, 2024

Haroon-Khel commented Jul 31, 2024 •

edited

Loading

adamfarley commented Nov 25, 2024

sxa commented Nov 25, 2024

sxa commented Nov 25, 2024 •

edited

Loading

sxa commented Nov 26, 2024

sxa commented Nov 29, 2024

sxa commented Dec 16, 2024

Haroon-Khel commented Feb 10, 2025 •

edited

Loading

Haroon-Khel commented Feb 10, 2025 •

edited

Loading

sxa commented Feb 10, 2025

Haroon-Khel commented Feb 11, 2025

Haroon-Khel commented Feb 11, 2025

Haroon-Khel commented Feb 11, 2025

Haroon-Khel commented Feb 17, 2025 •

edited

Loading

Haroon-Khel commented Feb 17, 2025 •

edited

Loading

Haroon-Khel commented Feb 18, 2025 •

edited

Loading

Unexpected tar error while un tarring jdk17 binary in ppc64le and arm32 ubuntu 2404 docker image #3547

Unexpected tar error while un tarring jdk17 binary in ppc64le and arm32 ubuntu 2404 docker image #3547

Comments

Haroon-Khel commented May 2, 2024

Haroon-Khel commented May 2, 2024

sxa commented May 3, 2024

sxa commented May 7, 2024 • edited Loading

sxa commented May 7, 2024 • edited Loading

adamfarley commented Jul 8, 2024

Haroon-Khel commented Jul 31, 2024 • edited Loading

adamfarley commented Nov 25, 2024

sxa commented Nov 25, 2024

sxa commented Nov 25, 2024 • edited Loading

sxa commented Nov 26, 2024

sxa commented Nov 29, 2024

sxa commented Dec 16, 2024

Haroon-Khel commented Feb 10, 2025 • edited Loading

Haroon-Khel commented Feb 10, 2025 • edited Loading

sxa commented Feb 10, 2025

Haroon-Khel commented Feb 11, 2025

Haroon-Khel commented Feb 11, 2025

Haroon-Khel commented Feb 11, 2025

Haroon-Khel commented Feb 17, 2025 • edited Loading

Haroon-Khel commented Feb 17, 2025 • edited Loading

Haroon-Khel commented Feb 18, 2025 • edited Loading

sxa commented May 7, 2024 •

edited

Loading

sxa commented May 7, 2024 •

edited

Loading

Haroon-Khel commented Jul 31, 2024 •

edited

Loading

sxa commented Nov 25, 2024 •

edited

Loading

Haroon-Khel commented Feb 10, 2025 •

edited

Loading

Haroon-Khel commented Feb 10, 2025 •

edited

Loading

Haroon-Khel commented Feb 17, 2025 •

edited

Loading

Haroon-Khel commented Feb 17, 2025 •

edited

Loading

Haroon-Khel commented Feb 18, 2025 •

edited

Loading