Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SIGSEGV when cache-from and cache-to are different locations in the same S3 bucket #5584

Open
parisBrentas opened this issue Dec 10, 2024 · 2 comments
Open

SIGSEGV when cache-from and cache-to are different locations in the same S3 bucket #5584

parisBrentas opened this issue Dec 10, 2024 · 2 comments
Labels
kind/bug

Comments

@parisBrentas
Copy link

parisBrentas commented Dec 10, 2024 •
edited
Loading

Running a build command where the cache-from and cache-to point to different locations in the same S3 bucket and the cache-from location contains a complete match causes build to fail with a SIGSEGV.

I've included what I believe is the relevant information below. Please let me know if you need anything else.

Command used

Below is the command used, with the env and project names changed

/usr/bin/docker buildx build --build-arg AGENT_VERSION=7.56.2 --cache-from type=s3,blobs_prefix=cache/docker/env-a/project-a/,manifests_prefix=cache/docker/env-a/project-a/,region=eu-west-2,bucket=gha-cache --cache-to type=s3,blobs_prefix=cache/docker/env-b/project-a/,manifests_prefix=cache/docker/env-b/project-a/,region=eu-west-2,bucket=gha-cache,mode=max --file dockerfiles/project-a/Dockerfile --iidfile /tmp/docker-build-push-SVUVoD/iidfile --tag 123456789.dkr.ecr.eu-west-2.amazonaws.com/services/project-a:7.56.2 --load --metadata-file /tmp/docker-build-push-SVUVoD/metadata-file .

My actual build uses two cache-from directives, one for env-a and one for env-b. This is just a minimum reproduction.

Environment

The build happens in Github Actions ubuntu-latest.
Buildkit version is v0.18.1.

Here is the debug output from the docker/setup-buildx-action@v3 action. 
::group::Docker info
Docker info
  ##[debug]Exec.exec: docker version
  /usr/bin/docker version
  Client: Docker Engine - Community
   Version:           26.1.3
   API version:       1.45
   Go version:        go1.21.10
   Git commit:        b72abbb
   Built:             Thu May 16 08:33:29 2024
   OS/Arch:           linux/amd64
   Context:           default
  
  Server: Docker Engine - Community
   Engine:
    Version:          26.1.3
    API version:      1.45 (minimum version 1.24)
    Go version:       go1.21.10
    Git commit:       8e96db1
    Built:            Thu May 16 08:33:29 2024
    OS/Arch:          linux/amd64
    Experimental:     false
   containerd:
    Version:          1.7.24
    GitCommit:        88bf19b2105c8b17560993bee28a01ddc2f97182
   runc:
    Version:          1.2.2
    GitCommit:        v1.2.2-0-g7cb3632
   docker-init:
    Version:          0.19.0
    GitCommit:        de40ad0
  ##[debug]Exec.exec: docker info
  /usr/bin/docker info
  Client: Docker Engine - Community
   Version:    26.1.3
   Context:    default
   Debug Mode: false
   Plugins:
    buildx: Docker Buildx (Docker Inc.)
      Version:  v0.19.1
      Path:     /usr/libexec/docker/cli-plugins/docker-buildx
    compose: Docker Compose (Docker Inc.)
      Version:  v2.27.1
      Path:     /usr/libexec/docker/cli-plugins/docker-compose
  
  Server:
   Containers: 0
    Running: 0
    Paused: 0
    Stopped: 0
   Images: 15
   Server Version: 26.1.3
   Storage Driver: overlay2
    Backing Filesystem: extfs
    Supports d_type: true
    Using metacopy: false
    Native Overlay Diff: false
    userxattr: false
   Logging Driver: json-file
   Cgroup Driver: systemd
   Cgroup Version: 2
   Plugins:
    Volume: local
    Network: bridge host ipvlan macvlan null overlay
    Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
   Swarm: inactive
   Runtimes: io.containerd.runc.v2 runc
   Default Runtime: runc
   Init Binary: docker-init
   containerd version: 88bf19b2105c8b17560993bee28a01ddc2f97182
   runc version: v1.2.2-0-g7cb3632
   init version: de40ad0
   Security Options:
    apparmor
    seccomp
     Profile: builtin
    cgroupns
   Kernel Version: 6.5.0-1025-azure
   Operating System: Ubuntu 22.04.5 LTS
   OSType: linux
   Architecture: x86_64
   CPUs: 4
   Total Memory: 15.61GiB
   Name: runner
   ID: d292b4cc-acf7-406d-b50f-82233a76aea4
   Docker Root Dir: /var/lib/docker
   Debug Mode: false
   Username: githubactions
   Experimental: false
   Insecure Registries:
    127.0.0.0/8
   Live Restore Enabled: false
  
  ::endgroup::
##[debug]Docker.isAvailable ok: /usr/bin/docker
##[debug]Buildx.isStandalone: false
##[debug]Exec.getExecOutput: docker buildx
##[debug]Buildx.isAvailable: true
::group::Buildx version
Buildx version
  ##[debug]Docker.isAvailable ok: /usr/bin/docker
  ##[debug]Buildx.isStandalone: false
  ##[debug]Exec.exec: docker buildx version
  /usr/bin/docker buildx version
  github.com/docker/buildx v0.19.1 5113f9ea89afd597302961caf99cecd142c7ba05
  ::endgroup::
##[debug]Exec.getExecOutput: docker context inspect --format {{.Name}}
::group::Inspecting default docker context
Inspecting default docker context
  ##[debug]Exec.getExecOutput: docker context inspect --format=json default
  [
    {
      "Name": "default",
      "Metadata": {},
      "Endpoints": {
        "docker": {
          "Host": "unix:///var/run/docker.sock",
          "SkipTLSVerify": false
        }
      },
      "TLSMaterial": {},
      "Storage": {
        "MetadataPath": "<IN MEMORY>",
        "TLSPath": "<IN MEMORY>"
      }
    }
  ]
  ::endgroup::
::group::Creating a new builder instance
Creating a new builder instance
  ##[debug]Docker.isAvailable ok: /usr/bin/docker
  ##[debug]Buildx.isStandalone: false
  ##[debug]Exec.getExecOutput: docker buildx version
  ##[debug]Buildx.versionSatisfies 0.19.1 statisfies >=0.3.0: true
  ##[debug]Docker.isAvailable ok: /usr/bin/docker
  ##[debug]Buildx.isStandalone: false
  ##[debug]Exec.getExecOutput: docker buildx create --name builder-4ad53ab7-9202-4911-a131-36db5f768a15 --driver docker-container --buildkitd-flags --allow-insecure-entitlement security.insecure --allow-insecure-entitlement network.host --use
  /usr/bin/docker buildx create --name builder-4ad53ab7-9202-4911-a131-36db5f768a15 --driver docker-container --buildkitd-flags --allow-insecure-entitlement security.insecure --allow-insecure-entitlement network.host --use
  builder-4ad53ab7-9202-4911-a131-36db5f768a15
  ::endgroup::
::group::Booting builder
Booting builder
  ##[debug]Buildx.versionSatisfies 0.19.1 statisfies >=0.4.0: true
  ##[debug]Docker.isAvailable ok: /usr/bin/docker
  ##[debug]Buildx.isStandalone: false
  ##[debug]Exec.getExecOutput: docker buildx inspect --bootstrap --builder builder-4ad53ab7-9202-4911-a131-36db5f768a15
  /usr/bin/docker buildx inspect --bootstrap --builder builder-4ad53ab7-9202-4911-a131-36db5f768a15
  #1 [internal] booting buildkit
  #1 pulling image moby/buildkit:buildx-stable-1
  #1 pulling image moby/buildkit:buildx-stable-1 5.2s done
  #1 creating container buildx_buildkit_builder-4ad53ab7-9202-4911-a131-36db5f768a150
  #1 creating container buildx_buildkit_builder-4ad53ab7-9202-4911-a131-36db5f768a150 1.2s done
  #1 DONE 6.4s
  Name:          builder-4ad53ab7-9202-4911-a131-36db5f768a15
  Driver:        docker-container
  Last Activity: 2024-12-10 11:06:32 +0000 UTC
  
  Nodes:
  Name:                  builder-4ad53ab7-9202-4911-a131-36db5f768a150
  Endpoint:              unix:///var/run/docker.sock
  Status:                running
  BuildKit daemon flags: --allow-insecure-entitlement security.insecure --allow-insecure-entitlement network.host
  BuildKit version:      v0.18.1
  Platforms:             linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
  Labels:
   org.mobyproject.buildkit.worker.executor:         oci
   org.mobyproject.buildkit.worker.hostname:         323be59ce4fc
   org.mobyproject.buildkit.worker.network:          host
   org.mobyproject.buildkit.worker.oci.process-mode: sandbox
   org.mobyproject.buildkit.worker.selinux.enabled:  false
   org.mobyproject.buildkit.worker.snapshotter:      overlayfs
  GC Policy rule#0:
   All:            false
   Filters:        type==source.local,type==exec.cachemount,type==source.git.checkout
   Keep Duration:  48h0m0s
   Max Used Space: 488.3MiB
  GC Policy rule#1:
   All:            false
   Keep Duration:  1440h0m0s
   Reserved Space: 9.313GiB
   Max Used Space: 93.13GiB
   Min Free Space: 27.94GiB
  GC Policy rule#2:
   All:            false
   Reserved Space: 9.313GiB
   Max Used Space: 93.13GiB
   Min Free Space: 27.94GiB
  GC Policy rule#3:
   All:            true
   Reserved Space: 9.313GiB
   Max Used Space: 93.13GiB
   Min Free Space: 27.94GiB
  ::endgroup::
##[debug]Docker.isAvailable ok: /usr/bin/docker
##[debug]Buildx.isStandalone: false
##[debug]Exec.getExecOutput: docker buildx inspect builder-4ad53ab7-9202-4911-a131-36db5f768a15
::group::Inspect builder
Inspect builder
  {
    "nodes": [
      {
        "name": "builder-4ad53ab7-9202-4911-a131-36db5f768a150",
        "endpoint": "unix:///var/run/docker.sock",
        "status": "running",
        "buildkitd-flags": "--allow-insecure-entitlement security.insecure --allow-insecure-entitlement network.host",
        "buildkit": "v0.18.1",
        "platforms": "linux/amd64,linux/amd64/v2,linux/amd64/v3,linux/386",
        "features": {
          "Automatically load images to the Docker Engine image store": true,
          "Cache export": true,
          "Docker exporter": true,
          "Multi-platform build": true,
          "OCI exporter": true
        },
        "labels": {
          "org.mobyproject.buildkit.worker.executor": "oci",
          "org.mobyproject.buildkit.worker.hostname": "323be59ce4fc",
          "org.mobyproject.buildkit.worker.network": "host",
          "org.mobyproject.buildkit.worker.oci.process-mode": "sandbox",
          "org.mobyproject.buildkit.worker.selinux.enabled": "false",
          "org.mobyproject.buildkit.worker.snapshotter": "overlayfs"
        },
        "gcPolicy": [
          {
            "all": false,
            "filter": [
              "type==source.local",
              "type==exec.cachemount",
              "type==source.git.checkout"
            ],
            "keepDuration": "48h0m0s"
          },
          {
            "all": false,
            "keepDuration": "1440h0m0s"
          },
          {
            "all": false
          },
          {
            "all": true
          }
        ]
      }
    ],
    "name": "builder-4ad53ab7-9202-4911-a131-36db5f768a15",
    "driver": "docker-container",
    "lastActivity": "2024-12-10T11:06:32.000Z"
  }
  ::endgroup::
::group::BuildKit version
BuildKit version
  builder-4ad53ab7-9202-4911-a131-36db5f768a150: v0.18.1
  ::endgroup::
##[debug]Node Action run completed with exit code 0
##[debug]Save intra-action state isPost = true
##[debug]Save intra-action state cleanup = true
##[debug]Save intra-action state standalone = false
##[debug]Save intra-action state builderName = builder-4ad53ab7-9202-4911-a131-36db5f768a15
##[debug]Save intra-action state builderDriver = docker-container
##[debug]Save intra-action state certsDir = /home/runner/.docker/buildx/certs
##[debug]Save intra-action state containerName = buildx_buildkit_builder-4ad53ab7-9202-4911-a131-36db5f768a150
##[debug]Save intra-action state isDebug = true
##[debug]Set output name = builder-4ad53ab7-9202-4911-a131-36db5f768a15
##[debug]Set output driver = docker-container
##[debug]Set output platforms = linux/amd64,linux/amd64/v2,linux/amd64/v3,linux/386
##[debug]Set output nodes = [
##[debug]  {
##[debug]    "name": "builder-4ad53ab7-9202-4911-a131-36db5f768a150",
##[debug]    "endpoint": "unix:///var/run/docker.sock",
##[debug]    "status": "running",
##[debug]    "buildkitd-flags": "--allow-insecure-entitlement security.insecure --allow-insecure-entitlement network.host",
##[debug]    "buildkit": "v0.18.1",
##[debug]    "platforms": "linux/amd64,linux/amd64/v2,linux/amd64/v3,linux/386",
##[debug]    "features": {
##[debug]      "Automatically load images to the Docker Engine image store": true,
##[debug]      "Cache export": true,
##[debug]      "Docker exporter": true,
##[debug]      "Multi-platform build": true,
##[debug]      "OCI exporter": true
##[debug]    },
##[debug]    "labels": {
##[debug]      "org.mobyproject.buildkit.worker.executor": "oci",
##[debug]      "org.mobyproject.buildkit.worker.hostname": "323be59ce4fc",
##[debug]      "org.mobyproject.buildkit.worker.network": "host",
##[debug]      "org.mobyproject.buildkit.worker.oci.process-mode": "sandbox",
##[debug]      "org.mobyproject.buildkit.worker.selinux.enabled": "false",
##[debug]      "org.mobyproject.buildkit.worker.snapshotter": "overlayfs"
##[debug]    },
##[debug]    "gcPolicy": [
##[debug]      {
##[debug]        "all": false,
##[debug]        "filter": [
##[debug]          "type==source.local",
##[debug]          "type==exec.cachemount",
##[debug]          "type==source.git.checkout"
##[debug]        ],
##[debug]        "keepDuration": "48h0m0s"
##[debug]      },
##[debug]      {
##[debug]        "all": false,
##[debug]        "keepDuration": "1440h0m0s"
##[debug]      },
##[debug]      {
##[debug]        "all": false
##[debug]      },
##[debug]      {
##[debug]        "all": true
##[debug]      }
##[debug]    ]
##[debug]  }
##[debug]]
##[debug]Set output endpoint = unix:///var/run/docker.sock
##[debug]Set output status = running
##[debug]Set output flags = --allow-insecure-entitlement security.insecure --allow-insecure-entitlement network.host
##[debug]Finishing: Setup Docker buildx

Logs

The error is

panic: runtime error: invalid memory address or nil pointer dereference
  [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x132332b]

and it gets emitted from

github.com/moby/buildkit/cache/remotecache/s3.(*readerAtCloser).ReadAt(0xc003523dc0, {0xc003d14000, 0x8000, 0x8000}, 0xc000)
  	/src/cache/remotecache/s3/readerat.go:52 +0x1cb

The cache-to location in the bucket contains some of the layers but no manifest file. My guess would be that the connection to the bucket closes after the cache is read but before the cache can finish being written.

Here's the buildkit logs 
##[debug]Exec.getExecOutput: docker logs buildx_buildkit_builder-4ad53ab7-9202-4911-a131-36db5f768a150
  /usr/bin/docker logs buildx_buildkit_builder-4ad53ab7-9202-4911-a131-36db5f768a150
  time="2024-12-10T11:06:39Z" level=info msg="auto snapshotter: using overlayfs"
  time="2024-12-10T11:06:39Z" level=warning msg="using host network as the default"
  time="2024-12-10T11:06:39Z" level=info msg="found worker \"9u1if5c61ym0e9e4vi5yaafwp\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:323be59ce4fc org.mobyproject.buildkit.worker.network:host org.mobyproject.buildkit.worker.oci.process-mode:sandbox org.mobyproject.buildkit.worker.selinux.enabled:false org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/amd64/v2 linux/amd64/v3 linux/386]"
  time="2024-12-10T11:06:39Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
  time="2024-12-10T11:06:39Z" level=info msg="found 1 workers, default=\"9u1if5c61ym0e9e4vi5yaafwp\""
  time="2024-12-10T11:06:39Z" level=warning msg="currently, only the default worker can be used."
  time="2024-12-10T11:06:39Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
  panic: runtime error: invalid memory address or nil pointer dereference
  [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x132332b]
  
  goroutine 22972 [running]:
  github.com/moby/buildkit/cache/remotecache/s3.(*readerAtCloser).ReadAt(0xc003523dc0, {0xc003d14000, 0x8000, 0x8000}, 0xc000)
  	/src/cache/remotecache/s3/readerat.go:52 +0x1cb
  io.(*SectionReader).ReadAt(0x45834?, {0xc003d14000?, 0xc001958408?, 0xc003dc9800?}, 0x0?)
  	/usr/local/go/src/io/io.go:555 +0x93
  io.(*SectionReader).Read(0xc003569320, {0xc003d14000?, 0xc003d14000?, 0x8000?})
  	/usr/local/go/src/io/io.go:516 +0x4f
  github.com/aws/smithy-go/transport/http/internal/io.(*safeReadCloser).Read(0x477dc9?, {0xc003d14000?, 0xc003dc9938?, 0xc003dc9928?})
  	/src/vendor/github.com/aws/smithy-go/transport/http/internal/io/safe.go:59 +0x117
  io.(*LimitedReader).Read(0xc003804f48, {0xc003d14000?, 0x8000?, 0x8000?})
  	/usr/local/go/src/io/io.go:479 +0x43
  io.copyBuffer({0x7f12a5f9ae90, 0xc002d81508}, {0x1ed3220, 0xc003804f48}, {0x0, 0x0, 0x0})
  	/usr/local/go/src/io/io.go:429 +0x191
  io.Copy(...)
  	/usr/local/go/src/io/io.go:388
  net/http.persistConnWriter.ReadFrom({0x2be4920?}, {0x1ed3220?, 0xc003804f48?})
  	/usr/local/go/src/net/http/transport.go:1899 +0x5b
  bufio.(*Writer).ReadFrom(0xc0036fcec0, {0x1ed3220, 0xc003804f48})
  	/usr/local/go/src/bufio/bufio.go:794 +0x17c
  io.copyBuffer({0x1ed3520, 0xc0036fcec0}, {0x1ed3220, 0xc003804f48}, {0xc003cc6000, 0x8000, 0x8000})
  	/usr/local/go/src/io/io.go:415 +0x151
  io.CopyBuffer({0x1ed3520?, 0xc0036fcec0?}, {0x1ed3220?, 0xc003804f48?}, {0xc003cc6000?, 0x1992460?, 0xc000b32e08?})
  	/usr/local/go/src/io/io.go:402 +0x36
  net/http.(*transferWriter).doBodyCopy(0xc00357efa0, {0x1ed3520, 0xc0036fcec0}, {0x1ed3220, 0xc003804f48})
  	/usr/local/go/src/net/http/transfer.go:416 +0xe5
  net/http.(*transferWriter).writeBody(0xc00357efa0, {0x1ed3520, 0xc0036fcec0})
  	/usr/local/go/src/net/http/transfer.go:371 +0x366
  net/http.(*Request).write(0xc003811a40, {0x1ed3520, 0xc0036fcec0}, 0x0, 0x0, 0x0)
  	/usr/local/go/src/net/http/request.go:771 +0xaed
  net/http.(*persistConn).writeLoop(0xc002e38fc0)
  	/usr/local/go/src/net/http/transport.go:2522 +0x174
  created by net/http.(*Transport).dialConn in goroutine 23010
  	/usr/local/go/src/net/http/transport.go:1875 +0x15a5
  time="2024-12-10T11:09:17Z" level=info msg="auto snapshotter: using overlayfs"
  time="2024-12-10T11:09:17Z" level=warning msg="using host network as the default"
  time="2024-12-10T11:09:17Z" level=info msg="found worker \"9u1if5c61ym0e9e4vi5yaafwp\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:323be59ce4fc org.mobyproject.buildkit.worker.network:host org.mobyproject.buildkit.worker.oci.process-mode:sandbox org.mobyproject.buildkit.worker.selinux.enabled:false org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/amd64/v2 linux/amd64/v3 linux/386]"
  time="2024-12-10T11:09:17Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
  time="2024-12-10T11:09:17Z" level=info msg="found 1 workers, default=\"9u1if5c61ym0e9e4vi5yaafwp\""
  time="2024-12-10T11:09:17Z" level=warning msg="currently, only the default worker can be used."
  time="2024-12-10T11:09:17Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
  ::endgroup::
Here's the relevant parts from the build process 
::group::Docker info

[command]/usr/bin/docker version
Client: Docker Engine - Community
 Version:           26.1.3
 API version:       1.45
 Go version:        go1.21.10
 Git commit:        b72abbb
 Built:             Thu May 16 08:33:29 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.1.3
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.10
  Git commit:       8e96db1
  Built:            Thu May 16 08:33:29 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.24
  GitCommit:        88bf19b2105c8b17560993bee28a01ddc2f97182
 runc:
  Version:          1.2.2
  GitCommit:        v1.2.2-0-g7cb3632
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
[command]/usr/bin/docker info
Client: Docker Engine - Community
 Version:    26.1.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.19.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 16
 Server Version: 26.1.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 88bf19b2105c8b17560993bee28a01ddc2f97182
 runc version: v1.2.2-0-g7cb3632
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.5.0-1025-azure
 Operating System: Ubuntu 22.04.5 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.61GiB
 Name: runner
 ID: d292b4cc-acf7-406d-b50f-82233a76aea4
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: githubactions
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

::endgroup::

::group::Buildx version

[command]/usr/bin/docker buildx version
github.com/docker/buildx v0.19.1 5113f9ea89afd597302961caf99cecd142c7ba05
::endgroup::

[command]/usr/bin/docker buildx build --build-arg AGENT_VERSION=7.56.2 --cache-from type=s3,blobs_prefix=cache/docker/env-a/project-a/,manifests_prefix=cache/docker/env-a/project-a/,region=eu-west-2,bucket=gha-cache --cache-to type=s3,blobs_prefix=cache/docker/env-b/project-a/,manifests_prefix=cache/docker/env-b/project-a/,region=eu-west-2,bucket=gha-cache,mode=max --file dockerfiles/project-a/Dockerfile --iidfile /tmp/docker-build-push-SVUVoD/iidfile --tag 123456789.dkr.ecr.eu-west-2.amazonaws.com/services/project-a:7.56.2 --load --metadata-file /tmp/docker-build-push-SVUVoD/metadata-file .
#0 building with "builder-4ad53ab7-9202-4911-a131-36db5f768a15" instance using docker-container driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 2.28kB done
#1 DONE 0.0s

#2 [internal] load metadata for gcr.io/parent/image:7.56.2
#2 DONE 0.9s

#3 [internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s

#4 [internal] load build context
#4 DONE 0.0s

#5 [ 1/12] FROM gcr.io/parent/image:7.56.2@sha256:ca9e192f56e3d67fe7b34702353af02f2fcb277474bd3bb9fd3ecd8eab4f15d3
#5 resolve gcr.io/parent/image:7.56.2@sha256:ca9e192f56e3d67fe7b34702353af02f2fcb277474bd3bb9fd3ecd8eab4f15d3 done
#5 DONE 0.0s

#6 importing cache manifest from s3:14979828866107435508
#6 DONE 0.5s

#4 [internal] load build context
#4 transferring context: 8.33kB done
#4 DONE 0.0s

#7 [10/12] RUN stuff
#7 CACHED

#8 [ 8/12] RUN stuff
#8 CACHED

#9 [11/12] RUN stuff
#9 CACHED

#10 [ 2/12] RUN stuff
#10 CACHED

#11 [ 4/12] RUN stuff
#11 CACHED

#12 [ 7/12] RUN stuff
#12 CACHED

#13 [ 9/12] RUN stuff
#13 CACHED

#14 [ 3/12] RUN stuff
#14 CACHED

#15 [ 5/12] RUN stuff
#15 CACHED

#16 [ 6/12] RUN stuff
#16 CACHED

#17 [12/12] COPY dockerfiles/project-a/file-a.py /opt/project-a/
#17 CACHED

#18 exporting to docker image format
#18 exporting layers done
#18 exporting manifest sha256:fa3cc393564a0bcd1ed2f0f7e461c21eb785210fbb079f895a6b656b5ecc6f53 done
#18 exporting config sha256:9546d8fe74b953587db33a8eb6fe2337eb08ed44a4cb4f17cf7e7983873020aa done
#18 ...

#17 [12/12] COPY dockerfiles/project-a/file-a.py /opt/project-a/
#17 sha256:38f756e9bbef4ff86559a06921c74d5f86b9f591ab19636839096ca4dfe3132a 2.23kB / 2.23kB 0.2s done
#17 sha256:b4456234ac7df654e281dd5ba31e38abb8120e6d4613b6728d2748953dc004af 80.74MB / 436.24MB 5.1s
#17 sha256:915e6beb805233dedb6a7f8570a555ebcd4f38878110546ce734c3742325fdaf 30.44MB / 30.44MB 2.9s done
#17 sha256:97496254bad4d6cfb74ef974bd3cbd1e575efd85faf49fcd9bc0b76495bad418 12.12MB / 12.12MB 1.9s done
#17 sha256:f0a23eb3910760a2a9fd9185abdb0dfff8e40f84bd0ed317767d3fe429d0d143 65.67MB / 65.67MB 4.8s done
#17 sha256:b7b518d8f014082ee6055096e05b9de77d1933af8382c501a7d617a7e3c39b75 62.91MB / 315.04MB 3.3s
#17 sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 32B / 32B 0.1s done
#17 sha256:095c6ebf2f90fb58a4fb1409945a15f10d7da7b7e13fc7468f6829f9de6ec03e 10.34kB / 10.34kB 0.2s done
#17 sha256:57f1e7573c21731802726662d8d02cb0c90ad7a48e7f208fc154a4d44a0a27fd 33.55MB / 46.30MB 2.0s
#17 sha256:5d4be21b6fd17b3184e26643e5efe92826387b36e5a220278cce69321d336f07 294B / 294B 0.2s done
#17 sha256:57f1e7573c21731802726662d8d02cb0c90ad7a48e7f208fc154a4d44a0a27fd 36.70MB / 46.30MB 2.1s
#17 sha256:e862906c3ef8dabb360d7d9d7443c789d4996b91aa45e2915e7236c02f0030f5 328B / 328B 0.2s done
#17 sha256:57f1e7573c21731802726662d8d02cb0c90ad7a48e7f208fc154a4d44a0a27fd 39.85MB / 46.30MB 2.3s
#17 sha256:57f1e7573c21731802726662d8d02cb0c90ad7a48e7f208fc154a4d44a0a27fd 42.99MB / 46.30MB 2.4s
#17 sha256:162f1c73efff49019eb8564f1d80fb8e39fc839fe9a38d485c4af02df4516e26 0B / 5.55MB 0.2s
#17 sha256:57f1e7573c21731802726662d8d02cb0c90ad7a48e7f208fc154a4d44a0a27fd 46.30MB / 46.30MB 2.6s
#17 sha256:162f1c73efff49019eb8564f1d80fb8e39fc839fe9a38d485c4af02df4516e26 2.10MB / 5.55MB 0.3s
#17 sha256:57f1e7573c21731802726662d8d02cb0c90ad7a48e7f208fc154a4d44a0a27fd 46.30MB / 46.30MB 2.6s done
#17 sha256:162f1c73efff49019eb8564f1d80fb8e39fc839fe9a38d485c4af02df4516e26 5.55MB / 5.55MB 0.5s done
#17 sha256:b7b518d8f014082ee6055096e05b9de77d1933af8382c501a7d617a7e3c39b75 79.69MB / 315.04MB 4.1s
#17 sha256:b4456234ac7df654e281dd5ba31e38abb8120e6d4613b6728d2748953dc004af 102.76MB / 436.24MB 6.2s
#17 sha256:b7b518d8f014082ee6055096e05b9de77d1933af8382c501a7d617a7e3c39b75 98.57MB / 315.04MB 5.0s
#17 sha256:b4456234ac7df654e281dd5ba31e38abb8120e6d4613b6728d2748953dc004af 126.88MB / 436.24MB 7.2s
#17 sha256:b7b518d8f014082ee6055096e05b9de77d1933af8382c501a7d617a7e3c39b75 115.43MB / 315.04MB 5.7s
#17 sha256:b4456234ac7df654e281dd5ba31e38abb8120e6d4613b6728d2748953dc004af 148.90MB / 436.24MB 8.3s
#17 sha256:b7b518d8f014082ee6055096e05b9de77d1933af8382c501a7d617a7e3c39b75 132.12MB / 315.04MB 6.5s
#17 sha256:b7b518d8f014082ee6055096e05b9de77d1933af8382c501a7d617a7e3c39b75 150.99MB / 315.04MB 7.4s
#17 sha256:b4456234ac7df654e281dd5ba31e38abb8120e6d4613b6728d2748953dc004af 171.97MB / 436.24MB 9.3s
#17 sha256:b7b518d8f014082ee6055096e05b9de77d1933af8382c501a7d617a7e3c39b75 167.77MB / 315.04MB 8.1s
#17 sha256:b4456234ac7df654e281dd5ba31e38abb8120e6d4613b6728d2748953dc004af 195.04MB / 436.24MB 10.4s
#17 sha256:b7b518d8f014082ee6055096e05b9de77d1933af8382c501a7d617a7e3c39b75 184.55MB / 315.04MB 8.9s
#17 sha256:b4456234ac7df654e281dd5ba31e38abb8120e6d4613b6728d2748953dc004af 218.10MB / 436.24MB 11.4s
#17 sha256:b7b518d8f014082ee6055096e05b9de77d1933af8382c501a7d617a7e3c39b75 201.33MB / 315.04MB 9.6s
#17 sha256:b7b518d8f014082ee6055096e05b9de77d1933af8382c501a7d617a7e3c39b75 220.20MB / 315.04MB 10.5s
#17 sha256:b4456234ac7df654e281dd5ba31e38abb8120e6d4613b6728d2748953dc004af 241.17MB / 436.24MB 12.5s
#17 sha256:b7b518d8f014082ee6055096e05b9de77d1933af8382c501a7d617a7e3c39b75 236.98MB / 315.04MB 11.3s
#17 sha256:b4456234ac7df654e281dd5ba31e38abb8120e6d4613b6728d2748953dc004af 264.24MB / 436.24MB 13.5s
#17 sha256:b7b518d8f014082ee6055096e05b9de77d1933af8382c501a7d617a7e3c39b75 253.76MB / 315.04MB 12.0s
#17 sha256:b4456234ac7df654e281dd5ba31e38abb8120e6d4613b6728d2748953dc004af 287.31MB / 436.24MB 14.6s
#17 sha256:b7b518d8f014082ee6055096e05b9de77d1933af8382c501a7d617a7e3c39b75 273.68MB / 315.04MB 12.9s
#17 sha256:b7b518d8f014082ee6055096e05b9de77d1933af8382c501a7d617a7e3c39b75 290.46MB / 315.04MB 13.7s
#17 sha256:b4456234ac7df654e281dd5ba31e38abb8120e6d4613b6728d2748953dc004af 311.43MB / 436.24MB 15.6s
#17 sha256:b7b518d8f014082ee6055096e05b9de77d1933af8382c501a7d617a7e3c39b75 309.69MB / 315.04MB 14.6s
#17 sha256:b4456234ac7df654e281dd5ba31e38abb8120e6d4613b6728d2748953dc004af 333.45MB / 436.24MB 16.7s
#17 sha256:b7b518d8f014082ee6055096e05b9de77d1933af8382c501a7d617a7e3c39b75 315.04MB / 315.04MB 14.8s done
#17 sha256:b4456234ac7df654e281dd5ba31e38abb8120e6d4613b6728d2748953dc004af 355.47MB / 436.24MB 17.6s
#17 sha256:b4456234ac7df654e281dd5ba31e38abb8120e6d4613b6728d2748953dc004af 377.49MB / 436.24MB 18.6s
#17 sha256:b4456234ac7df654e281dd5ba31e38abb8120e6d4613b6728d2748953dc004af 401.60MB / 436.24MB 19.7s
#17 sha256:b4456234ac7df654e281dd5ba31e38abb8120e6d4613b6728d2748953dc004af 423.62MB / 436.24MB 20.7s
#17 sha256:b4456234ac7df654e281dd5ba31e38abb8120e6d4613b6728d2748953dc004af 436.24MB / 436.24MB 21.2s done
#17 DONE 21.4s

#18 exporting to docker image format
#18 sending tarball
#18 ...

#19 importing to docker
#19 loading layer aff6671f35bd 557.06kB / 436.24MB
#19 loading layer aff6671f35bd 109.74MB / 436.24MB 2.1s
#19 loading layer aff6671f35bd 244.55MB / 436.24MB 4.1s
#19 loading layer aff6671f35bd 326.43MB / 436.24MB 6.2s
#19 loading layer aff6671f35bd 352.06MB / 436.24MB 8.2s
#19 loading layer aff6671f35bd 381.58MB / 436.24MB 10.3s
#19 loading layer 5657ff04fb27 327.68kB / 30.44MB
#19 loading layer c513da986341 131.07kB / 12.12MB
#19 loading layer 3bd4465631ac 10.34kB / 10.34kB
#19 loading layer 3c95e1948b2e 557.06kB / 65.67MB
#19 loading layer 3c95e1948b2e 60.72MB / 65.67MB 2.0s
#19 loading layer dbbcccf7bcca 557.06kB / 315.04MB
#19 loading layer dbbcccf7bcca 173.80MB / 315.04MB 2.0s
#19 loading layer dbbcccf7bcca 298.02MB / 315.04MB 4.2s
#19 loading layer 5f70bf18a086 32B / 32B
#19 loading layer f22d7a7c911d 491.52kB / 46.30MB
#19 loading layer f22d7a7c911d 38.34MB / 46.30MB 2.2s
#19 loading layer aa785246ba48 65.54kB / 5.55MB
#19 loading layer f76e7b8752b9 294B / 294B
#19 loading layer f22d7a7c911d 38.34MB / 46.30MB 3.5s done
#19 loading layer baaf811f3dc7 328B / 328B
#19 loading layer 886481f00645 2.23kB / 2.23kB
#19 loading layer aff6671f35bd 381.58MB / 436.24MB 29.9s done
#19 loading layer 5657ff04fb27 327.68kB / 30.44MB 18.1s done
#19 loading layer c513da986341 131.07kB / 12.12MB 15.9s done
#19 loading layer 3bd4465631ac 10.34kB / 10.34kB 15.5s done
#19 loading layer 3c95e1948b2e 60.72MB / 65.67MB 15.5s done
#19 loading layer dbbcccf7bcca 307.49MB / 315.04MB 12.9s done
#19 loading layer 5f70bf18a086 32B / 32B 3.6s done
#19 loading layer aa785246ba48 65.54kB / 5.55MB 0.5s done
#19 loading layer f76e7b8752b9 294B / 294B 0.1s done
#19 loading layer baaf811f3dc7 328B / 328B 0.1s done
#19 loading layer 886481f00645 2.23kB / 2.23kB 0.1s done
#19 DONE 29.9s

#18 exporting to docker image format
#18 sending tarball 34.7s done
#18 DONE 55.9s

#20 exporting cache to Amazon S3
#20 preparing build cache for export
#20 writing layer sha256:162f1c73efff49019eb8564f1d80fb8e39fc839fe9a38d485c4af02df4516e26
#20 writing layer sha256:57f1e7573c21731802726662d8d02cb0c90ad7a48e7f208fc154a4d44a0a27fd
#20 writing layer sha256:915e6beb805233dedb6a7f8570a555ebcd4f38878110546ce734c3742325fdaf
#20 writing layer sha256:97496254bad4d6cfb74ef974bd3cbd1e575efd85faf49fcd9bc0b76495bad418

ERROR: failed to receive status: rpc error: code = Unavailable desc = error reading from server: EOF
::group::Removing temp folder /tmp/docker-build-push-SVUVoD

::endgroup::
@tonistiigi
Copy link
Member

cc @bpaquet

bpaquet added a commit to bpaquet/buildkit that referenced this issue Dec 15, 2024
@bpaquet
[s3 cache] Fix upload of downloaded layer
a6dc2c3 
see moby#5584.

Seems this is a regression related to moby#4551, which happen when buildkit need to export a S3 layer directly from a downloaded S3 layer.

With the new wrapper, there is no exception, and buildkit download and re upload them without any issue. Checksum and size of layers are identical.
@bpaquet bpaquet mentioned this issue Dec 15, 2024
Open
bpaquet added a commit to bpaquet/buildkit that referenced this issue Dec 15, 2024
@bpaquet
[s3 cache] Fix upload of downloaded layer
2190c2f 
see moby#5584.

Seems this is a regression related to moby#4551, which happen when buildkit need to export a S3 layer directly from a downloaded S3 layer.

With the new wrapper, there is no exception, and buildkit download and re upload them without any issue. Checksum and size of layers are identical.
bpaquet added a commit to bpaquet/buildkit that referenced this issue Dec 15, 2024
@bpaquet
[s3 cache] Fix upload of downloaded layer
f04b18a 
see moby#5584.

Seems this is a regression related to moby#4551, which happen when buildkit need to export a S3 layer directly from a downloaded S3 layer.

With the new wrapper, there is no exception, and buildkit download and re upload them without any issue. Checksum and size of layers are identical.

Signed-off-by: Bertrand Paquet <[email protected]>
@bpaquet
Copy link
Contributor

bpaquet commented Dec 15, 2024

To reproduce it

Dockerfile

FROM debian:bookworm-slim

RUN apt-get update
  • build and cache the result into a directory a on s3
  • recreate your buildkit builder to restart from scratch
  • check there there is no directory b on s3
  • try to build by loading the cache from a and exporting it to b

@parisBrentas Thx for the report. FYI this is not really expected to use the same directory for manifests and blob, even if it should cause any trouble in practice :)

bpaquet added a commit to bpaquet/buildkit that referenced this issue Dec 17, 2024
@bpaquet
[s3 cache] Fix upload of downloaded layer
d990df6 
see moby#5584.

Seems this is a regression related to moby#4551, which happen when buildkit need to export a S3 layer directly from a downloaded S3 layer.

With the new wrapper, there is no exception, and buildkit download and re upload them without any issue. Checksum and size of layers are identical.

Signed-off-by: Bertrand Paquet <[email protected]>
bpaquet added a commit to bpaquet/buildkit that referenced this issue Jan 14, 2025
@bpaquet
[s3 cache] Fix upload of downloaded layer
6e252aa 
see moby#5584.

Seems this is a regression related to moby#4551, which happen when buildkit need to export a S3 layer directly from a downloaded S3 layer.

With the new wrapper, there is no exception, and buildkit download and re upload them without any issue. Checksum and size of layers are identical.

Signed-off-by: Bertrand Paquet <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@bpaquet @tonistiigi @parisBrentas