Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Aufs to overlay2 migration failure #301

Open
alexgg opened this issue Jun 23, 2022 · 1 comment · May be fixed by #303
Open

Aufs to overlay2 migration failure #301

alexgg opened this issue Jun 23, 2022 · 1 comment · May be fixed by #303
Assignees
@lmbarros @zoobot

Comments

@alexgg
Copy link
Contributor

alexgg commented Jun 23, 2022

Description

After a hostOS update on a balenaFin that performs an aufs to overlay migration, the engine in the newOS fails to perform the migration and errors with:

Jun 23 14:40:18 a5fa2e0 balenad[1995]: time="2022-06-23T14:40:18.955898073Z" level=info msg="Storage migration from aufs to overlay2 starting"
Jun 23 14:40:45 a5fa2e0 balenad[1995]: time="2022-06-23T14:40:45.998202208Z" level=info msg="transforming 342 layers(s) to overlay2"
Jun 23 14:40:46 a5fa2e0 balenad[1995]: time="2022-06-23T14:40:46.001704023Z" level=info msg="Storage migration finished, took 27.045128503s"
Jun 23 14:40:46 a5fa2e0 balenad[1995]: time="2022-06-23T14:40:46.001925741Z" level=error msg="failed aufs to overlay2 migration" error="Error moving layer data to overlay2: symlink hostname /var/lib/docker/overlay2.temp/00d0a01127ca0>
Jun 23 14:40:46 a5fa2e0 balenad[1995]: time="2022-06-23T14:40:46.002066418Z" level=warning msg="recovering from failed aufs to overlay migration" storage_root=/var/lib/docker
Jun 23 14:40:46 a5fa2e0 balenad[1995]: time="2022-06-23T14:40:46.237646393Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
Jun 23 14:40:46 a5fa2e0 balenad[1995]: time="2022-06-23T14:40:46.237842799Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
Jun 23 14:40:47 a5fa2e0 balenad[1995]: failed to start daemon: error cleaning up: Error migrating containers to aufs: Error rewriting container config for f6767d30d030f98813480bc41c26683d5ba91918331577767ca451a697f4920e: open /var/li>
Jun 23 14:40:47 a5fa2e0 systemd[1]: balena.service: Main process exited, code=exited, status=1/FAILURE

Steps to reproduce the issue:

  1. Perform a hostOS update from 2.72.0+rev1 to 2.95.8 on a balenaFin. Note that the error might need the exact application container images that this has been reproduced with.
  2. On boot, check that the engine has failed to migrate.

Describe the results you received:

The engine fails to starts and keeps trying to migrate in a loop.

Describe the results you expected:
One of:

  • The engine correctly migrates the application containers
  • The engine fails to migrate the containers, but it removes the old ones and re-downloads new ones.

Additional information you deem important (e.g. issue happens only occasionally):

The issue happens with specific applications.

Output of balena-engine version:

Client:
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.2
 Git commit:        73c78258302d94f9652da995af6f65a621fac918
 Built:             Wed Mar  2 18:29:36 2022
 OS/Arch:           linux/arm
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.2
  Git commit:       73c78258302d94f9652da995af6f65a621fac918
  Built:            Wed Mar  2 18:29:36 2022
  OS/Arch:          linux/arm
  Experimental:     true
 containerd:
  Version:          1.4.0+unknown
  GitCommit:        
 runc:
  Version:          spec: 1.0.2-dev
  GitCommit:        
 balena-engine-init:
  Version:          0.13.0
  GitCommit:        949e6fa-dirty

Output of balena-engine info:

Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 14
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: journald
 Cgroup Driver: systemd
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host null
  Log: journald json-file local
 Swarm: 
  NodeID: 
  Is Manager: false
  Node Address: 
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: balena-engine-init
 containerd version: 
 runc version: 
 init version: 949e6fa-dirty (expected: de40ad007797e)
 Kernel Version: 5.10.83-v7
 Operating System: balenaOS 2.95.8
 OSType: linux
 Architecture: armv7l
 CPUs: 4
 Total Memory: 970.7MiB
 Name: a5fa2e0
 ID: DVU5:EUK4:ZANG:C4IA:DHRR:ZZI5:WW4Y:RK6C:CJRJ:Q3RW:JNLX:S6LQ
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

Additional environment details (device type, OS, etc.):

BalenaFin running balenaOS 2.95.8
@jellyfish-bot
Copy link

[alexgg] This has attached https://jel.ly.fish/9aaf5f96-265c-4c52-9e59-074e973b53f8

@zoobot zoobot self-assigned this Nov 12, 2022
@zoobot zoobot linked a pull request Nov 12, 2022 that will close this issue
Ignore missing config.v2.json files on migration #303
Draft
@lmbarros lmbarros linked a pull request Dec 19, 2022 that will close this issue
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lmbarros lmbarros

@zoobot zoobot

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Ignore missing config.v2.json files on migration balena-os/balena-engine
4 participants
@alexgg @lmbarros @zoobot @jellyfish-bot