Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Containerd fails to start containers on RHEL 8 when installed in a RAM disk. #6814

Closed
agaxprex opened this issue Sep 12, 2024 · 2 comments
Closed

Containerd fails to start containers on RHEL 8 when installed in a RAM disk. #6814

agaxprex opened this issue Sep 12, 2024 · 2 comments

Comments

@agaxprex
Copy link

agaxprex commented Sep 12, 2024 •
edited
Loading

Environmental Info:
RKE2 Version:
rke2 version v1.29.0+rke2r1 (4fd30c2)
go version go1.21.5 X:boringcrypto

Node(s) CPU architecture, OS, and Version:
4.18.0-477.10.1.el8_8.x86_64

Cluster Configuration:
1 server, storage constrained, data-dir under ramdisk
SELinux Permissive mode enabled
cgroupsv1, non-hybrid mode enabled

Describe the bug:
Attempting to start an RKE2 cluster on a server with limited storage. I'm keeping the installation of RKE2 on a ramdisk, but the containerd logs seems unable to launch containers. Seems related to missing CNI plugins, and possibly SELinux interference even in Permissive mode.

Steps To Reproduce:

  • Create RAM disk, 20 GB in storage
    • mkdir /tmp/rke2disk
    • sudo mount -t tmpfs -o size=20GB rke2disk /tmp/rke2disk
  • Installed RKE2 using airgapped tarball install (not using RPMs):

---

write-kubeconfig-mode: 666
token: *****************
data-dir: /tmp/rke2disk/rke2
cni: ['multus', 'canal']
tls-san:
  - localhost
  - cluster.local
disable: ['rke2-ingress-nginx']
node-label:
  - cs=server
snapshotter: native
etcd_disable_snapshots: true
node-name: **********
selinux: true
debug: true
disable-cloud-controller: true

Expected behavior:
Containerd starts containers sometime after loading images.

Actual behavior:
Containerd simply loads images, but seems runc fails at starting containers. Notably the following errors.

time="2022-01-01T01:51:40.206727872-05:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-yellowtail,Uid:dd5acff53d45cce9522711ad34cb2359,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown"

Full containerd.log

time="2022-01-01T01:48:46.763181840-05:00" level=info msg="starting containerd" revision=64c566767b99d0bacf26fb6e2355775c413331c9 version=v1.7.11-k3s2
time="2022-01-01T01:48:46.809035174-05:00" level=info msg="loading plugin \"io.containerd.warning.v1.deprecations\"..." type=io.containerd.warning.v1
time="2022-01-01T01:48:46.809132200-05:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
time="2022-01-01T01:48:46.813789698-05:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: FATAL: Module aufs not found in directory /lib/modules/4.18.0-477.10.1.el8_8.x86_64\\n\"): skip plugin" type=io.containerd.snapshotter.v1
time="2022-01-01T01:48:46.813851517-05:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1
time="2022-01-01T01:48:46.814186690-05:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /tmp/rke2disk/rke2/agent/containerd/io.containerd.snapshotter.v1.btrfs (tmpfs) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2022-01-01T01:48:46.814215017-05:00" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
time="2022-01-01T01:48:46.814280745-05:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.blockfile\"..." type=io.containerd.snapshotter.v1
time="2022-01-01T01:48:46.814332439-05:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.blockfile\"..." error="no scratch file generator: skip plugin" type=io.containerd.snapshotter.v1
time="2022-01-01T01:48:46.814355392-05:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
time="2022-01-01T01:48:46.814408961-05:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
time="2022-01-01T01:48:46.814608630-05:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.fuse-overlayfs\"..." type=io.containerd.snapshotter.v1
time="2022-01-01T01:48:46.814670828-05:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.stargz\"..." type=io.containerd.snapshotter.v1
time="2022-01-01T01:48:46.815286170-05:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
time="2022-01-01T01:48:46.815318739-05:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
time="2022-01-01T01:48:46.815336977-05:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
time="2022-01-01T01:48:46.815524661-05:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /tmp/rke2disk/rke2/agent/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2022-01-01T01:48:46.815550343-05:00" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
time="2022-01-01T01:48:46.815596887-05:00" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
time="2022-01-01T01:48:46.815615816-05:00" level=info msg="metadata content store policy set" policy=shared
time="2022-01-01T01:48:46.815792596-05:00" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
time="2022-01-01T01:48:46.815824207-05:00" level=info msg="loading plugin \"io.containerd.event.v1.exchange\"..." type=io.containerd.event.v1
time="2022-01-01T01:48:46.815848270-05:00" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
time="2022-01-01T01:48:46.815894816-05:00" level=info msg="loading plugin \"io.containerd.lease.v1.manager\"..." type=io.containerd.lease.v1
time="2022-01-01T01:48:46.815922028-05:00" level=info msg="loading plugin \"io.containerd.nri.v1.nri\"..." type=io.containerd.nri.v1
time="2022-01-01T01:48:46.815943259-05:00" level=info msg="NRI interface is disabled by configuration."
time="2022-01-01T01:48:46.815966276-05:00" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
time="2022-01-01T01:48:46.816097099-05:00" level=error msg="cleanup working directory in namespace" error="open /tmp/rke2disk/rke2/agent/containerd/io.containerd.runtime.v2.task/k8s.io: no such file or directory" namespace=k8s.io
time="2022-01-01T01:48:46.816136887-05:00" level=info msg="loading plugin \"io.containerd.runtime.v2.shim\"..." type=io.containerd.runtime.v2
time="2022-01-01T01:48:46.816163777-05:00" level=info msg="loading plugin \"io.containerd.sandbox.store.v1.local\"..." type=io.containerd.sandbox.store.v1
time="2022-01-01T01:48:46.816187647-05:00" level=info msg="loading plugin \"io.containerd.sandbox.controller.v1.local\"..." type=io.containerd.sandbox.controller.v1
time="2022-01-01T01:48:46.816212639-05:00" level=info msg="loading plugin \"io.containerd.streaming.v1.manager\"..." type=io.containerd.streaming.v1
time="2022-01-01T01:48:46.816237710-05:00" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1
time="2022-01-01T01:48:46.816267424-05:00" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
time="2022-01-01T01:48:46.816294134-05:00" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
time="2022-01-01T01:48:46.816317519-05:00" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
time="2022-01-01T01:48:46.816343732-05:00" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
time="2022-01-01T01:48:46.816369417-05:00" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
time="2022-01-01T01:48:46.816394681-05:00" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
time="2022-01-01T01:48:46.816416274-05:00" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
time="2022-01-01T01:48:46.816478207-05:00" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
time="2022-01-01T01:48:46.817025941-05:00" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
time="2022-01-01T01:48:46.817067330-05:00" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
time="2022-01-01T01:48:46.817092130-05:00" level=info msg="loading plugin \"io.containerd.transfer.v1.local\"..." type=io.containerd.transfer.v1
time="2022-01-01T01:48:46.817129923-05:00" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
time="2022-01-01T01:48:46.817215294-05:00" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
time="2022-01-01T01:48:46.817244290-05:00" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
time="2022-01-01T01:48:46.817266790-05:00" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
time="2022-01-01T01:48:46.817289830-05:00" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
time="2022-01-01T01:48:46.817314876-05:00" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
time="2022-01-01T01:48:46.817340676-05:00" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
time="2022-01-01T01:48:46.817365236-05:00" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
time="2022-01-01T01:48:46.817387230-05:00" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
time="2022-01-01T01:48:46.817415077-05:00" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
time="2022-01-01T01:48:46.817515907-05:00" level=info msg="loading plugin \"io.containerd.grpc.v1.sandbox-controllers\"..." type=io.containerd.grpc.v1
time="2022-01-01T01:48:46.817543121-05:00" level=info msg="loading plugin \"io.containerd.grpc.v1.sandboxes\"..." type=io.containerd.grpc.v1
time="2022-01-01T01:48:46.817566377-05:00" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
time="2022-01-01T01:48:46.817594747-05:00" level=info msg="loading plugin \"io.containerd.grpc.v1.streaming\"..." type=io.containerd.grpc.v1
time="2022-01-01T01:48:46.817619157-05:00" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
time="2022-01-01T01:48:46.817645917-05:00" level=info msg="loading plugin \"io.containerd.grpc.v1.transfer\"..." type=io.containerd.grpc.v1
time="2022-01-01T01:48:46.817669356-05:00" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
time="2022-01-01T01:48:46.817690285-05:00" level=info msg="loading plugin \"io.containerd.grpc.v1.cri\"..." type=io.containerd.grpc.v1
time="2022-01-01T01:48:46.818169627-05:00" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:native DefaultRuntimeName:runc DefaultRuntime:{Type: Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] PrivilegedWithoutHostDevices:false PrivilegedWithoutHostDevicesAllDevicesAllowed:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0 Snapshotter: SandboxMode:} UntrustedWorkloadRuntime:{Type: Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] PrivilegedWithoutHostDevices:false PrivilegedWithoutHostDevicesAllDevicesAllowed:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0 Snapshotter: SandboxMode:} Runtimes:map[runc:{Type:io.containerd.runc.v2 Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[SystemdCgroup:true] PrivilegedWithoutHostDevices:false PrivilegedWithoutHostDevicesAllDevicesAllowed:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0 Snapshotter: SandboxMode:podsandbox}] NoPivot:false DisableSnapshotAnnotations:true DiscardUnpackedLayers:false IgnoreBlockIONotEnabledErrors:false IgnoreRdtNotEnabledErrors:false} CniConfig:{NetworkPluginBinDir:/opt/cni/bin NetworkPluginConfDir:/etc/cni/net.d NetworkPluginMaxConfNum:1 NetworkPluginSetupSerially:false NetworkPluginConfTemplate: IPPreference:} Registry:{ConfigPath: Mirrors:map[] Configs:map[] Auths:map[] Headers:map[]} ImageDecryption:{KeyModel:node} DisableTCPService:true StreamServerAddress:127.0.0.1 StreamServerPort:10010 StreamIdleTimeout:4h0m0s EnableSelinux:true SelinuxCategoryRange:1024 SandboxImage:index.docker.io/rancher/mirrored-pause:3.6 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} MaxContainerLogLineSize:16384 DisableCgroup:false DisableApparmor:false RestrictOOMScoreAdj:false MaxConcurrentDownloads:3 DisableProcMount:false UnsetSeccompProfile: TolerateMissingHugetlbController:true DisableHugetlbController:true DeviceOwnershipFromSecurityContext:false IgnoreImageDefinedVolumes:false NetNSMountsUnderStateDir:false EnableUnprivilegedPorts:true EnableUnprivilegedICMP:true EnableCDI:false CDISpecDirs:[/etc/cdi /var/run/cdi] ImagePullProgressTimeout:5m0s DrainExecSyncIOTimeout:0s} ContainerdRootDir:/tmp/rke2disk/rke2/agent/containerd ContainerdEndpoint:/run/k3s/containerd/containerd.sock RootDir:/tmp/rke2disk/rke2/agent/containerd/io.containerd.grpc.v1.cri StateDir:/run/k3s/containerd/io.containerd.grpc.v1.cri}"
time="2022-01-01T01:48:46.818272410-05:00" level=info msg="Connect containerd service"
time="2022-01-01T01:48:46.818315252-05:00" level=info msg="using legacy CRI server"
time="2022-01-01T01:48:46.818326576-05:00" level=info msg="using experimental NRI integration - disable nri plugin to prevent this"
time="2022-01-01T01:48:46.818372450-05:00" level=info msg="Get image filesystem path \"/tmp/rke2disk/rke2/agent/containerd/io.containerd.snapshotter.v1.native\""
time="2022-01-01T01:48:46.819189550-05:00" level=error msg="failed to load cni during init, please check CRI plugin status before setting up network for pods" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
time="2022-01-01T01:48:46.819266519-05:00" level=info msg="loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." type=io.containerd.tracing.processor.v1
time="2022-01-01T01:48:46.819296147-05:00" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
time="2022-01-01T01:48:46.819315801-05:00" level=info msg="loading plugin \"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
time="2022-01-01T01:48:46.819335530-05:00" level=info msg="skipping tracing processor initialization (no tracing plugin)" error="no OpenTelemetry endpoint: skip plugin"
time="2022-01-01T01:48:46.819492850-05:00" level=info msg="Start subscribing containerd event"
time="2022-01-01T01:48:46.819646926-05:00" level=info msg="Start recovering state"
time="2022-01-01T01:48:46.819866441-05:00" level=info msg=serving... address=/run/k3s/containerd/containerd.sock.ttrpc
time="2022-01-01T01:48:46.819875339-05:00" level=info msg="Start event monitor"
time="2022-01-01T01:48:46.819924590-05:00" level=info msg="Start snapshots syncer"
time="2022-01-01T01:48:46.819947589-05:00" level=info msg="Start cni network conf syncer for default"
time="2022-01-01T01:48:46.819960821-05:00" level=info msg=serving... address=/run/k3s/containerd/containerd.sock
time="2022-01-01T01:48:46.819963559-05:00" level=info msg="Start streaming server"
time="2022-01-01T01:48:46.820060810-05:00" level=info msg="containerd successfully booted in 0.058052s"
time="2022-01-01T01:49:07.809381639-05:00" level=info msg="ImageCreate event name:\"docker.io/rancher/hardened-calico:v3.26.3-build20231109\""
time="2022-01-01T01:49:07.813842630-05:00" level=info msg="ImageCreate event name:\"sha256:116d7534875a5767406cd0b844e8bb4c88193831c72d78ccf00abb00dc1bf652\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:49:07.815290101-05:00" level=info msg="ImageCreate event name:\"docker.io/rancher/hardened-flannel:v0.23.0-build20231109\""
time="2022-01-01T01:49:07.819495295-05:00" level=info msg="ImageUpdate event name:\"docker.io/rancher/hardened-calico:v3.26.3-build20231109\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:49:07.821035915-05:00" level=info msg="ImageCreate event name:\"sha256:c776826db2fda39152c467ecee8dd0d8f0414b1443423a2c819174f5d3bef7c1\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:49:07.822496909-05:00" level=info msg="ImageUpdate event name:\"docker.io/rancher/hardened-flannel:v0.23.0-build20231109\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.067050914-05:00" level=info msg="ImageCreate event name:\"docker.io/rancher/mirrored-pause:3.6\""
time="2022-01-01T01:50:35.069979019-05:00" level=info msg="ImageCreate event name:\"sha256:6270bb605e12e581514ada5fd5b3216f727db55dc87d5889c790e4c760683fee\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.070674594-05:00" level=info msg="ImageCreate event name:\"docker.io/rancher/nginx-ingress-controller:nginx-1.9.3-hardened1\""
time="2022-01-01T01:50:35.080237850-05:00" level=info msg="ImageUpdate event name:\"docker.io/rancher/mirrored-pause:3.6\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.082015354-05:00" level=info msg="ImageCreate event name:\"docker.io/rancher/mirrored-sig-storage-snapshot-controller:v6.2.1\""
time="2022-01-01T01:50:35.085573644-05:00" level=info msg="ImageCreate event name:\"docker.io/rancher/hardened-dns-node-cache:1.22.20-build20231010\""
time="2022-01-01T01:50:35.089155294-05:00" level=info msg="ImageCreate event name:\"sha256:bfdece8fa3f1449a2b25c12f3e375c57258a6cd4d925f7983177f5f652afc885\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.093461454-05:00" level=info msg="ImageCreate event name:\"docker.io/rancher/klipper-helm:v0.8.2-build20230815\""
time="2022-01-01T01:50:35.097322239-05:00" level=info msg="ImageCreate event name:\"docker.io/rancher/klipper-lb:v0.4.4\""
time="2022-01-01T01:50:35.100984344-05:00" level=info msg="ImageCreate event name:\"docker.io/rancher/hardened-kubernetes:v1.29.0-rke2r1-build20231213\""
time="2022-01-01T01:50:35.108594740-05:00" level=info msg="ImageUpdate event name:\"docker.io/rancher/nginx-ingress-controller:nginx-1.9.3-hardened1\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.111739244-05:00" level=info msg="ImageCreate event name:\"docker.io/rancher/hardened-etcd:v3.5.9-k3s1-build20230802\""
time="2022-01-01T01:50:35.114998189-05:00" level=info msg="ImageCreate event name:\"docker.io/rancher/hardened-k8s-metrics-server:v0.6.3-build20231009\""
time="2022-01-01T01:50:35.117608570-05:00" level=info msg="ImageCreate event name:\"docker.io/rancher/rke2-runtime:v1.29.0-rke2r1\""
time="2022-01-01T01:50:35.119285014-05:00" level=info msg="ImageCreate event name:\"sha256:1ef6c138bd5f2ac45f7b4ee54db0e513efad8576909ae9829ba649fb4b067388\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.120234595-05:00" level=info msg="ImageCreate event name:\"docker.io/rancher/hardened-coredns:v1.10.1-build20231009\""
time="2022-01-01T01:50:35.122720430-05:00" level=info msg="ImageUpdate event name:\"docker.io/rancher/mirrored-sig-storage-snapshot-controller:v6.2.1\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.123505074-05:00" level=info msg="ImageCreate event name:\"docker.io/rancher/rke2-cloud-provider:v1.28.2-build20231016\""
time="2022-01-01T01:50:35.126238220-05:00" level=info msg="ImageCreate event name:\"sha256:760c20ed3d8830c73e49f33e855b2c5c4a9ac37cac1a4965334ae875839f206c\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.127469779-05:00" level=info msg="ImageCreate event name:\"docker.io/rancher/hardened-cluster-autoscaler:v1.8.6-build20231009\""
time="2022-01-01T01:50:35.130039590-05:00" level=info msg="ImageUpdate event name:\"docker.io/rancher/hardened-dns-node-cache:1.22.20-build20231010\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.131849589-05:00" level=info msg="ImageCreate event name:\"docker.io/rancher/mirrored-ingress-nginx-kube-webhook-certgen:v20230312-helm-chart-4.5.2-28-g66a760794\""
time="2022-01-01T01:50:35.134186370-05:00" level=info msg="ImageCreate event name:\"docker.io/rancher/mirrored-sig-storage-snapshot-validation-webhook:v6.2.2\""
time="2022-01-01T01:50:35.137288624-05:00" level=info msg="ImageCreate event name:\"sha256:5f89cb8137ccbd39377d91b9d75faf4ec4ee0a2d2a3a63635535b10c69c935fa\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.138455065-05:00" level=info msg="ImageUpdate event name:\"docker.io/rancher/klipper-helm:v0.8.2-build20230815\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.139807700-05:00" level=info msg="ImageCreate event name:\"sha256:af74bd845c4a83b7e4fa48e0c5a91dcda8843f586794fbb8b7f4bb7ed9e8cc56\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.140864385-05:00" level=info msg="ImageUpdate event name:\"docker.io/rancher/klipper-lb:v0.4.4\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.141804659-05:00" level=info msg="ImageCreate event name:\"sha256:ab7fea56055ba0fa4a1840780b4d52ae7143945acf79dde1069edeeffa9461c6\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.143256080-05:00" level=info msg="ImageUpdate event name:\"docker.io/rancher/hardened-kubernetes:v1.29.0-rke2r1-build20231213\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.144808494-05:00" level=info msg="ImageCreate event name:\"sha256:c6b7a4f2f79b24f9310e769ce7c1e0caba47fbf2d03a2025b19bee2090dae94d\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.145937084-05:00" level=info msg="ImageUpdate event name:\"docker.io/rancher/hardened-etcd:v3.5.9-k3s1-build20230802\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.146926250-05:00" level=info msg="ImageCreate event name:\"sha256:9051c01da7c50cf471beced1c737e13984760ddaf7288ab52f852a7009e95246\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.147887675-05:00" level=info msg="ImageUpdate event name:\"docker.io/rancher/hardened-k8s-metrics-server:v0.6.3-build20231009\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.149189690-05:00" level=info msg="ImageCreate event name:\"sha256:4f24500e130755055472a1d5d0cf3d76c782d3ac86dce1bcb45d037a8b31a3f5\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.150626800-05:00" level=info msg="ImageUpdate event name:\"docker.io/rancher/rke2-runtime:v1.29.0-rke2r1\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.152195315-05:00" level=info msg="ImageCreate event name:\"sha256:99513a1cabd6fe947a3df0eb03f3d22ef8fbfba91e555d78959eefd20201de65\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.154428480-05:00" level=info msg="ImageUpdate event name:\"docker.io/rancher/hardened-coredns:v1.10.1-build20231009\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.156983850-05:00" level=info msg="ImageCreate event name:\"sha256:00bad7e3c2263e3ef9e6231031a05929d9e2a56c6b87c89ad8cae6cc8ef8befe\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.158692314-05:00" level=info msg="ImageUpdate event name:\"docker.io/rancher/rke2-cloud-provider:v1.28.2-build20231016\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.160299644-05:00" level=info msg="ImageCreate event name:\"sha256:1b5aedc400f2a025e0bb0eae8ff29226ea759831d02c1afce4dababc607a9c74\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.161909850-05:00" level=info msg="ImageUpdate event name:\"docker.io/rancher/hardened-cluster-autoscaler:v1.8.6-build20231009\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.163861310-05:00" level=info msg="ImageCreate event name:\"sha256:5a86b03a88d2316e2317c2576449a95ddbd105d69b2fe7b01d667b0ebab37422\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.165131020-05:00" level=info msg="ImageUpdate event name:\"docker.io/rancher/mirrored-ingress-nginx-kube-webhook-certgen:v20230312-helm-chart-4.5.2-28-g66a760794\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.166397985-05:00" level=info msg="ImageCreate event name:\"sha256:ff52c2bcf9f8893ac479bade578b25e9f4315173bcba6f605ca94a4c7ab84235\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:50:35.168451939-05:00" level=info msg="ImageUpdate event name:\"docker.io/rancher/mirrored-sig-storage-snapshot-validation-webhook:v6.2.2\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:51:08.640771485-05:00" level=info msg="ImageCreate event name:\"docker.io/rancher/hardened-sriov-network-device-plugin:v3.5.1-build20231009\""
time="2022-01-01T01:51:08.644288804-05:00" level=info msg="ImageCreate event name:\"sha256:0c4930c72d24b2c07147549d4b119eeb5f2f4e8a46dfede9a116b0019f68329c\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:51:08.645805804-05:00" level=info msg="ImageCreate event name:\"docker.io/rancher/hardened-ib-sriov-cni:v1.0.2-build20231009\""
time="2022-01-01T01:51:08.649470650-05:00" level=info msg="ImageUpdate event name:\"docker.io/rancher/hardened-sriov-network-device-plugin:v3.5.1-build20231009\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:51:08.650764144-05:00" level=info msg="ImageCreate event name:\"docker.io/rancher/hardened-sriov-network-resources-injector:v1.5-build20231009\""
time="2022-01-01T01:51:08.655099979-05:00" level=info msg="ImageCreate event name:\"docker.io/rancher/hardened-whereabouts:v0.6.2-build20231009\""
time="2022-01-01T01:51:08.658685599-05:00" level=info msg="ImageCreate event name:\"sha256:1cb4ae9f10098f6073d2ae72c0cb9c95fb323dae436db51129a71140546422b6\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:51:08.660104760-05:00" level=info msg="ImageCreate event name:\"docker.io/rancher/hardened-multus-cni:v4.0.2-build20231009\""
time="2022-01-01T01:51:08.664595939-05:00" level=info msg="ImageUpdate event name:\"docker.io/rancher/hardened-ib-sriov-cni:v1.0.2-build20231009\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:51:08.666179424-05:00" level=info msg="ImageCreate event name:\"docker.io/rancher/hardened-cni-plugins:v1.2.0-build20231009\""
time="2022-01-01T01:51:08.670423520-05:00" level=info msg="ImageCreate event name:\"docker.io/rancher/hardened-sriov-network-operator:v1.2.0-build20231010\""
time="2022-01-01T01:51:08.674360635-05:00" level=info msg="ImageCreate event name:\"sha256:fc13f6f3c12237b733c46221e93a3209dfc06cdb161fb344fa608531ca970671\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:51:08.676212954-05:00" level=info msg="ImageCreate event name:\"docker.io/rancher/hardened-sriov-network-config-daemon:v1.2.0-build20230607\""
time="2022-01-01T01:51:08.679975554-05:00" level=info msg="ImageCreate event name:\"docker.io/rancher/hardened-sriov-cni:v2.6.3-build20231009\""
time="2022-01-01T01:51:08.683410829-05:00" level=info msg="ImageUpdate event name:\"docker.io/rancher/hardened-sriov-network-resources-injector:v1.5-build20231009\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:51:08.684884315-05:00" level=info msg="ImageCreate event name:\"docker.io/rancher/hardened-sriov-network-webhook:v1.2.0-build20230607\""
time="2022-01-01T01:51:08.688005420-05:00" level=info msg="ImageCreate event name:\"sha256:638b76afb5d053df336a9543703d78d6359f739ca3f353be63f981f50fa14bdb\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:51:08.689359914-05:00" level=info msg="ImageUpdate event name:\"docker.io/rancher/hardened-whereabouts:v0.6.2-build20231009\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:51:08.690633954-05:00" level=info msg="ImageCreate event name:\"sha256:6fe45ed7141bb1fb8ee685ea52f99ad5e65cb026a18fe0ae617479b26060b34c\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:51:08.692404375-05:00" level=info msg="ImageUpdate event name:\"docker.io/rancher/hardened-multus-cni:v4.0.2-build20231009\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:51:08.694454952-05:00" level=info msg="ImageCreate event name:\"sha256:6fbf05a8266e71d40f4fab9b9137a7ff99bceec4bf1869cebc74d68f1e9e6716\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:51:08.695715205-05:00" level=info msg="ImageUpdate event name:\"docker.io/rancher/hardened-cni-plugins:v1.2.0-build20231009\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:51:08.697129994-05:00" level=info msg="ImageCreate event name:\"sha256:2288c1727ebf9ed7f0e61cf523ca6e10591f0b4ebbc3726e94ef829144c5c58c\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:51:08.698421275-05:00" level=info msg="ImageUpdate event name:\"docker.io/rancher/hardened-sriov-network-operator:v1.2.0-build20231010\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:51:08.699565655-05:00" level=info msg="ImageCreate event name:\"sha256:96a9602db90782133ac1d980e35c9a46f45e641133e3809be9cff52db917f2b9\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:51:08.700929109-05:00" level=info msg="ImageUpdate event name:\"docker.io/rancher/hardened-sriov-network-config-daemon:v1.2.0-build20230607\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:51:08.702368070-05:00" level=info msg="ImageCreate event name:\"sha256:19e4169d6e8ad7d0d3ebcd8025f0f2a7501ccb043fa83fdbc8af098ef66e423b\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:51:08.703961459-05:00" level=info msg="ImageUpdate event name:\"docker.io/rancher/hardened-sriov-cni:v2.6.3-build20231009\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:51:08.705335218-05:00" level=info msg="ImageCreate event name:\"sha256:ac20acf2e2d75196a999b6fa36f71c1f47f9948f4802680ad98b82f0069c48d5\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:51:08.706480444-05:00" level=info msg="ImageUpdate event name:\"docker.io/rancher/hardened-sriov-network-webhook:v1.2.0-build20230607\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"
time="2022-01-01T01:51:09.330435640-05:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-yellowtail,Uid:dd5acff53d45cce9522711ad34cb2359,Namespace:kube-system,Attempt:0,}"
time="2022-01-01T01:51:09.433069320-05:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2022-01-01T01:51:09.433245999-05:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2022-01-01T01:51:09.433885025-05:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2022-01-01T01:51:09.434198430-05:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2022-01-01T01:51:09.598933135-05:00" level=info msg="shim disconnected" id=13a70e25b1d9b77b2947dd470a82a614faf808bd67a6340b67c6c564449d1e08 namespace=k8s.io
time="2022-01-01T01:51:09.599083274-05:00" level=warning msg="cleaning up after shim disconnected" id=13a70e25b1d9b77b2947dd470a82a614faf808bd67a6340b67c6c564449d1e08 namespace=k8s.io
time="2022-01-01T01:51:09.599120380-05:00" level=info msg="cleaning up dead shim" namespace=k8s.io
time="2022-01-01T01:51:09.670589460-05:00" level=warning msg="cleanup warnings time=\"2022-01-01T01:51:09-05:00\" level=warning msg=\"failed to read init pid file\" error=\"open /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/13a70e25b1d9b77b2947dd470a82a614faf808bd67a6340b67c6c564449d1e08/init.pid: no such file or directory\" runtime=io.containerd.runc.v2\n" namespace=k8s.io
time="2022-01-01T01:51:09.671463639-05:00" level=error msg="copy shim log" error="read /proc/self/fd/19: file already closed" namespace=k8s.io
time="2022-01-01T01:51:09.682924936-05:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-yellowtail,Uid:dd5acff53d45cce9522711ad34cb2359,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown"
time="2022-01-01T01:51:24.901261099-05:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-yellowtail,Uid:dd5acff53d45cce9522711ad34cb2359,Namespace:kube-system,Attempt:0,}"
time="2022-01-01T01:51:24.978289630-05:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2022-01-01T01:51:24.979391200-05:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2022-01-01T01:51:24.980378744-05:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2022-01-01T01:51:24.980428085-05:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2022-01-01T01:51:25.145790405-05:00" level=info msg="shim disconnected" id=7ed830ac6dcef3fc09e92b9bad12a60f9deda032a9ffb27b083b4d68f07134d3 namespace=k8s.io
time="2022-01-01T01:51:25.145936075-05:00" level=warning msg="cleaning up after shim disconnected" id=7ed830ac6dcef3fc09e92b9bad12a60f9deda032a9ffb27b083b4d68f07134d3 namespace=k8s.io
time="2022-01-01T01:51:25.145974570-05:00" level=info msg="cleaning up dead shim" namespace=k8s.io
time="2022-01-01T01:51:25.216480455-05:00" level=warning msg="cleanup warnings time=\"2022-01-01T01:51:25-05:00\" level=warning msg=\"failed to read init pid file\" error=\"open /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/7ed830ac6dcef3fc09e92b9bad12a60f9deda032a9ffb27b083b4d68f07134d3/init.pid: no such file or directory\" runtime=io.containerd.runc.v2\n" namespace=k8s.io
time="2022-01-01T01:51:25.217345310-05:00" level=error msg="copy shim log" error="read /proc/self/fd/19: file already closed" namespace=k8s.io
time="2022-01-01T01:51:25.227885184-05:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-yellowtail,Uid:dd5acff53d45cce9522711ad34cb2359,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown"
time="2022-01-01T01:51:39.900727390-05:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-yellowtail,Uid:dd5acff53d45cce9522711ad34cb2359,Namespace:kube-system,Attempt:0,}"
time="2022-01-01T01:51:39.981310175-05:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2022-01-01T01:51:39.981532524-05:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2022-01-01T01:51:39.981590710-05:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2022-01-01T01:51:39.981633859-05:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2022-01-01T01:51:40.126636584-05:00" level=info msg="shim disconnected" id=98c52edaaa5a753804348485d32b9962f554ff8d5962d17efe0b7d09bf6e81e4 namespace=k8s.io
time="2022-01-01T01:51:40.126801105-05:00" level=warning msg="cleaning up after shim disconnected" id=98c52edaaa5a753804348485d32b9962f554ff8d5962d17efe0b7d09bf6e81e4 namespace=k8s.io
time="2022-01-01T01:51:40.126838090-05:00" level=info msg="cleaning up dead shim" namespace=k8s.io
time="2022-01-01T01:51:40.198369930-05:00" level=warning msg="cleanup warnings time=\"2022-01-01T01:51:40-05:00\" level=warning msg=\"failed to read init pid file\" error=\"open /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/98c52edaaa5a753804348485d32b9962f554ff8d5962d17efe0b7d09bf6e81e4/init.pid: no such file or directory\" runtime=io.containerd.runc.v2\n" namespace=k8s.io
time="2022-01-01T01:51:40.199192759-05:00" level=error msg="copy shim log" error="read /proc/self/fd/19: file already closed" namespace=k8s.io
time="2022-01-01T01:51:40.206727872-05:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-yellowtail,Uid:dd5acff53d45cce9522711ad34cb2359,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown"
time="2022-01-01T01:51:51.900862880-05:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-yellowtail,Uid:dd5acff53d45cce9522711ad34cb2359,Namespace:kube-system,Attempt:0,}"
time="2022-01-01T01:51:51.988615114-05:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2022-01-01T01:51:51.988835235-05:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2022-01-01T01:51:51.988878994-05:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2022-01-01T01:51:51.988911984-05:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2022-01-01T01:51:52.128612239-05:00" level=info msg="shim disconnected" id=7a4c8c304f72062d11944f771d27ca4e89c891a9616cb18fe41ff7f90ce700f2 namespace=k8s.io
time="2022-01-01T01:51:52.128747175-05:00" level=warning msg="cleaning up after shim disconnected" id=7a4c8c304f72062d11944f771d27ca4e89c891a9616cb18fe41ff7f90ce700f2 namespace=k8s.io
time="2022-01-01T01:51:52.128794394-05:00" level=info msg="cleaning up dead shim" namespace=k8s.io
time="2022-01-01T01:51:52.201135480-05:00" level=warning msg="cleanup warnings time=\"2022-01-01T01:51:52-05:00\" level=warning msg=\"failed to read init pid file\" error=\"open /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/7a4c8c304f72062d11944f771d27ca4e89c891a9616cb18fe41ff7f90ce700f2/init.pid: no such file or directory\" runtime=io.containerd.runc.v2\n" namespace=k8s.io
time="2022-01-01T01:51:52.202019999-05:00" level=error msg="copy shim log" error="read /proc/self/fd/19: file already closed" namespace=k8s.io
time="2022-01-01T01:51:52.212916545-05:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-yellowtail,Uid:dd5acff53d45cce9522711ad34cb2359,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown"
time="2022-01-01T01:52:03.900425560-05:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-yellowtail,Uid:dd5acff53d45cce9522711ad34cb2359,Namespace:kube-system,Attempt:0,}"
time="2022-01-01T01:52:03.980658984-05:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2022-01-01T01:52:03.981930615-05:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2022-01-01T01:52:03.982082479-05:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2022-01-01T01:52:03.983057170-05:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2022-01-01T01:52:04.127366850-05:00" level=info msg="shim disconnected" id=691627fe1f365f45bd797504ead4ca427cb0d5a06d0c4cffc2cb1dbe51a00ebf namespace=k8s.io
time="2022-01-01T01:52:04.127483825-05:00" level=warning msg="cleaning up after shim disconnected" id=691627fe1f365f45bd797504ead4ca427cb0d5a06d0c4cffc2cb1dbe51a00ebf namespace=k8s.io
time="2022-01-01T01:52:04.127517135-05:00" level=info msg="cleaning up dead shim" namespace=k8s.io
time="2022-01-01T01:52:04.195686920-05:00" level=warning msg="cleanup warnings time=\"2022-01-01T01:52:04-05:00\" level=warning msg=\"failed to read init pid file\" error=\"open /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/691627fe1f365f45bd797504ead4ca427cb0d5a06d0c4cffc2cb1dbe51a00ebf/init.pid: no such file or directory\" runtime=io.containerd.runc.v2\n" namespace=k8s.io
time="2022-01-01T01:52:04.196502584-05:00" level=error msg="copy shim log" error="read /proc/self/fd/19: file already closed" namespace=k8s.io
time="2022-01-01T01:52:04.207800429-05:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-yellowtail,Uid:dd5acff53d45cce9522711ad34cb2359,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown"
time="2022-01-01T01:52:16.901452684-05:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-yellowtail,Uid:dd5acff53d45cce9522711ad34cb2359,Namespace:kube-system,Attempt:0,}"
time="2022-01-01T01:52:16.983735734-05:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2022-01-01T01:52:16.983976684-05:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2022-01-01T01:52:16.984021565-05:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2022-01-01T01:52:16.984053370-05:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2022-01-01T01:52:17.133772290-05:00" level=info msg="shim disconnected" id=e27a0406823c10d67c40dd24502dc2cde735f6859a5c46f2f9649d73024e6c55 namespace=k8s.io
time="2022-01-01T01:52:17.133902504-05:00" level=warning msg="cleaning up after shim disconnected" id=e27a0406823c10d67c40dd24502dc2cde735f6859a5c46f2f9649d73024e6c55 namespace=k8s.io
time="2022-01-01T01:52:17.133929154-05:00" level=info msg="cleaning up dead shim" namespace=k8s.io
time="2022-01-01T01:52:17.206921124-05:00" level=warning msg="cleanup warnings time=\"2022-01-01T01:52:17-05:00\" level=warning msg=\"failed to read init pid file\" error=\"open /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/e27a0406823c10d67c40dd24502dc2cde735f6859a5c46f2f9649d73024e6c55/init.pid: no such file or directory\" runtime=io.containerd.runc.v2\n" namespace=k8s.io
time="2022-01-01T01:52:17.207731835-05:00" level=error msg="copy shim log" error="read /proc/self/fd/19: file already closed" namespace=k8s.io
time="2022-01-01T01:52:17.218411745-05:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-yellowtail,Uid:dd5acff53d45cce9522711ad34cb2359,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown"
time="2022-01-01T01:52:30.900744317-05:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-yellowtail,Uid:dd5acff53d45cce9522711ad34cb2359,Namespace:kube-system,Attempt:0,}"
time="2022-01-01T01:52:30.981711005-05:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2022-01-01T01:52:30.982795272-05:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2022-01-01T01:52:30.982847109-05:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2022-01-01T01:52:30.982860629-05:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2022-01-01T01:52:31.129216024-05:00" level=info msg="shim disconnected" id=8f6c66fd57962c98ae5bfbee479efcefee780671db83704e748126bf70e6f2db namespace=k8s.io
time="2022-01-01T01:52:31.129364360-05:00" level=warning msg="cleaning up after shim disconnected" id=8f6c66fd57962c98ae5bfbee479efcefee780671db83704e748126bf70e6f2db namespace=k8s.io
time="2022-01-01T01:52:31.129401084-05:00" level=info msg="cleaning up dead shim" namespace=k8s.io
time="2022-01-01T01:52:31.204671349-05:00" level=warning msg="cleanup warnings time=\"2022-01-01T01:52:31-05:00\" level=warning msg=\"failed to read init pid file\" error=\"open /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/8f6c66fd57962c98ae5bfbee479efcefee780671db83704e748126bf70e6f2db/init.pid: no such file or directory\" runtime=io.containerd.runc.v2\n" namespace=k8s.io
time="2022-01-01T01:52:31.205498499-05:00" level=error msg="copy shim log" error="read /proc/self/fd/19: file already closed" namespace=k8s.io
time="2022-01-01T01:52:31.218831334-05:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-yellowtail,Uid:dd5acff53d45cce9522711ad34cb2359,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown"
time="2022-01-01T01:52:43.900735410-05:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-yellowtail,Uid:dd5acff53d45cce9522711ad34cb2359,Namespace:kube-system,Attempt:0,}"
time="2022-01-01T01:52:43.980386674-05:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2022-01-01T01:52:43.980569765-05:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2022-01-01T01:52:43.980609820-05:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2022-01-01T01:52:43.980647924-05:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2022-01-01T01:52:44.139446661-05:00" level=info msg="shim disconnected" id=b6f5a3e77fd444edbdf3656d282b3491c76e7aa1fbefd439a7341c1026a80f10 namespace=k8s.io
time="2022-01-01T01:52:44.139534668-05:00" level=warning msg="cleaning up after shim disconnected" id=b6f5a3e77fd444edbdf3656d282b3491c76e7aa1fbefd439a7341c1026a80f10 namespace=k8s.io
time="2022-01-01T01:52:44.139550986-05:00" level=info msg="cleaning up dead shim" namespace=k8s.io
time="2022-01-01T01:52:44.198501595-05:00" level=warning msg="cleanup warnings time=\"2022-01-01T01:52:44-05:00\" level=warning msg=\"failed to read init pid file\" error=\"open /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/b6f5a3e77fd444edbdf3656d282b3491c76e7aa1fbefd439a7341c1026a80f10/init.pid: no such file or directory\" runtime=io.containerd.runc.v2\n" namespace=k8s.io
time="2022-01-01T01:52:44.199379559-05:00" level=error msg="copy shim log" error="read /proc/self/fd/19: file already closed" namespace=k8s.io
time="2022-01-01T01:52:44.211860410-05:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-yellowtail,Uid:dd5acff53d45cce9522711ad34cb2359,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown"
time="2022-01-01T01:52:56.900683764-05:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-yellowtail,Uid:dd5acff53d45cce9522711ad34cb2359,Namespace:kube-system,Attempt:0,}"
time="2022-01-01T01:52:56.983153815-05:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2022-01-01T01:52:56.983377879-05:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2022-01-01T01:52:56.983449264-05:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2022-01-01T01:52:56.983497490-05:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2022-01-01T01:52:57.141893586-05:00" level=info msg="shim disconnected" id=17622db43fcb1cd27b0c22f881c43803a9cb24a7fb7ab33f2081ee4f0405ecda namespace=k8s.io
time="2022-01-01T01:52:57.142021890-05:00" level=warning msg="cleaning up after shim disconnected" id=17622db43fcb1cd27b0c22f881c43803a9cb24a7fb7ab33f2081ee4f0405ecda namespace=k8s.io
time="2022-01-01T01:52:57.142066044-05:00" level=info msg="cleaning up dead shim" namespace=k8s.io
time="2022-01-01T01:52:57.216267379-05:00" level=warning msg="cleanup warnings time=\"2022-01-01T01:52:57-05:00\" level=warning msg=\"failed to read init pid file\" error=\"open /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/17622db43fcb1cd27b0c22f881c43803a9cb24a7fb7ab33f2081ee4f0405ecda/init.pid: no such file or directory\" runtime=io.containerd.runc.v2\n" namespace=k8s.io
time="2022-01-01T01:52:57.217086545-05:00" level=error msg="copy shim log" error="read /proc/self/fd/19: file already closed" namespace=k8s.io
time="2022-01-01T01:52:57.227885339-05:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-yellowtail,Uid:dd5acff53d45cce9522711ad34cb2359,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown"
time="2022-01-01T01:53:09.900418200-05:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-yellowtail,Uid:dd5acff53d45cce9522711ad34cb2359,Namespace:kube-system,Attempt:0,}"
time="2022-01-01T01:53:09.990718725-05:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2022-01-01T01:53:09.991773805-05:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2022-01-01T01:53:09.992903175-05:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2022-01-01T01:53:09.993010234-05:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2022-01-01T01:53:10.170963418-05:00" level=info msg="shim disconnected" id=af5f903d5c5c95a66d5a932274b5dc1c63bf2083e21cc464870bfc159aad6a34 namespace=k8s.io
time="2022-01-01T01:53:10.171086029-05:00" level=warning msg="cleaning up after shim disconnected" id=af5f903d5c5c95a66d5a932274b5dc1c63bf2083e21cc464870bfc159aad6a34 namespace=k8s.io
time="2022-01-01T01:53:10.171111490-05:00" level=info msg="cleaning up dead shim" namespace=k8s.io
time="2022-01-01T01:53:10.242830370-05:00" level=warning msg="cleanup warnings time=\"2022-01-01T01:53:10-05:00\" level=warning msg=\"failed to read init pid file\" error=\"open /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/af5f903d5c5c95a66d5a932274b5dc1c63bf2083e21cc464870bfc159aad6a34/init.pid: no such file or directory\" runtime=io.containerd.runc.v2\n" namespace=k8s.io
time="2022-01-01T01:53:10.243640584-05:00" level=error msg="copy shim log" error="read /proc/self/fd/19: file already closed" namespace=k8s.io
time="2022-01-01T01:53:10.254752864-05:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-yellowtail,Uid:dd5acff53d45cce9522711ad34cb2359,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown"
time="2022-01-01T01:53:24.901474520-05:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-yellowtail,Uid:dd5acff53d45cce9522711ad34cb2359,Namespace:kube-system,Attempt:0,}"
time="2022-01-01T01:53:24.982989884-05:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2022-01-01T01:53:24.983185595-05:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2022-01-01T01:53:24.983229090-05:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2022-01-01T01:53:24.983267124-05:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2022-01-01T01:53:25.145098034-05:00" level=info msg="shim disconnected" id=d46c9f44dc0b5b3f18999c9445cf6f01c6f19c6de76d66c0ae2fd67c298861b8 namespace=k8s.io
time="2022-01-01T01:53:25.145218220-05:00" level=warning msg="cleaning up after shim disconnected" id=d46c9f44dc0b5b3f18999c9445cf6f01c6f19c6de76d66c0ae2fd67c298861b8 namespace=k8s.io
time="2022-01-01T01:53:25.145243269-05:00" level=info msg="cleaning up dead shim" namespace=k8s.io
time="2022-01-01T01:53:25.217199640-05:00" level=warning msg="cleanup warnings time=\"2022-01-01T01:53:25-05:00\" level=warning msg=\"failed to read init pid file\" error=\"open /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/d46c9f44dc0b5b3f18999c9445cf6f01c6f19c6de76d66c0ae2fd67c298861b8/init.pid: no such file or directory\" runtime=io.containerd.runc.v2\n" namespace=k8s.io
time="2022-01-01T01:53:25.218094189-05:00" level=error msg="copy shim log" error="read /proc/self/fd/19: file already closed" namespace=k8s.io
time="2022-01-01T01:53:25.229850520-05:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-yellowtail,Uid:dd5acff53d45cce9522711ad34cb2359,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown"
time="2022-01-01T01:53:38.900978124-05:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-yellowtail,Uid:dd5acff53d45cce9522711ad34cb2359,Namespace:kube-system,Attempt:0,}"
time="2022-01-01T01:53:38.988649724-05:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2022-01-01T01:53:38.988863765-05:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2022-01-01T01:53:38.988905960-05:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2022-01-01T01:53:38.988935570-05:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2022-01-01T01:53:39.137700886-05:00" level=info msg="shim disconnected" id=f8eb51a2287b8af37ecd479b1c104a8b539b1efcb6afff09363652b3b40115f2 namespace=k8s.io
time="2022-01-01T01:53:39.137843490-05:00" level=warning msg="cleaning up after shim disconnected" id=f8eb51a2287b8af37ecd479b1c104a8b539b1efcb6afff09363652b3b40115f2 namespace=k8s.io
time="2022-01-01T01:53:39.137869444-05:00" level=info msg="cleaning up dead shim" namespace=k8s.io
time="2022-01-01T01:53:39.212458650-05:00" level=warning msg="cleanup warnings time=\"2022-01-01T01:53:39-05:00\" level=warning msg=\"failed to read init pid file\" error=\"open /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/f8eb51a2287b8af37ecd479b1c104a8b539b1efcb6afff09363652b3b40115f2/init.pid: no such file or directory\" runtime=io.containerd.runc.v2\n" namespace=k8s.io
time="2022-01-01T01:53:39.213274755-05:00" level=error msg="copy shim log" error="read /proc/self/fd/19: file already closed" namespace=k8s.io
time="2022-01-01T01:53:39.229805629-05:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-yellowtail,Uid:dd5acff53d45cce9522711ad34cb2359,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown"
time="2022-01-01T01:53:51.900849305-05:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-yellowtail,Uid:dd5acff53d45cce9522711ad34cb2359,Namespace:kube-system,Attempt:0,}"
time="2022-01-01T01:53:51.980800514-05:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2022-01-01T01:53:51.980991554-05:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2022-01-01T01:53:51.981033584-05:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2022-01-01T01:53:51.981069029-05:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2022-01-01T01:53:52.134349662-05:00" level=info msg="shim disconnected" id=012f8bd9a7cd67fbb71ecb5d08718f7f12c42f9a8278f9535d7745ac0c621a4c namespace=k8s.io
time="2022-01-01T01:53:52.134443638-05:00" level=warning msg="cleaning up after shim disconnected" id=012f8bd9a7cd67fbb71ecb5d08718f7f12c42f9a8278f9535d7745ac0c621a4c namespace=k8s.io
time="2022-01-01T01:53:52.134459574-05:00" level=info msg="cleaning up dead shim" namespace=k8s.io
time="2022-01-01T01:53:52.208476010-05:00" level=warning msg="cleanup warnings time=\"2022-01-01T01:53:52-05:00\" level=warning msg=\"failed to read init pid file\" error=\"open /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/012f8bd9a7cd67fbb71ecb5d08718f7f12c42f9a8278f9535d7745ac0c621a4c/init.pid: no such file or directory\" runtime=io.containerd.runc.v2\n" namespace=k8s.io
time="2022-01-01T01:53:52.209311888-05:00" level=error msg="copy shim log" error="read /proc/self/fd/19: file already closed" namespace=k8s.io
time="2022-01-01T01:53:52.226752900-05:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-yellowtail,Uid:dd5acff53d45cce9522711ad34cb2359,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown"
@brandond
Copy link
Member

brandond commented Sep 12, 2024 •
edited
Loading

data-dir under ramdisk
mount -t tmpfs -o size=20GB rke2disk /tmp/rke2disk

I don't think you can use overlayfs on top of tmpfs? I would ask at https://github.com/containerd/containerd if you want to be sure, but as far as I can tell this is not a problem with RKE2 - rather you are doing something that isn't supported by linux and/or containerd.

@brandond
Copy link
Member

brandond commented Sep 12, 2024 •
edited
Loading

Issues with tmpfs/overlayfs aside, this could also be related to selinux - see rancher/rke2-selinux#24.

Did you mount the tmpfs volume there before installing RKE2 and the rke2-selinux RPM? How are you persisting the selinux contexts across reboots? Does tmpfs even support the required selinux xattrs?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@brandond @agaxprex