[Release-1.25] - [BUG] rke2 initial auto ETCD snapshot created with 0B data, unable to restore to snapshot

[brandond]

Backport fix for [BUG] rke2 initial auto ETCD snapshot created with 0B data, unable to restore to snapshot

• [BUG] rke2 initial auto ETCD snapshot created with 0B data, unable to restore to snapshot #5008

[ShylajaDevadiga]

Validated fix using rke2 version v1.25.16-rc1+rke2r1

Environment Details

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:
Ubuntu 22.04

Cluster Configuration:
Single node

Config.yaml:

cat /etc/rancher/k3s/config,yaml
write-kubeconfig-mode: "0644"

Steps to reproduce the issue and validate the fix

1. Copy config.yaml
2. Install rke2
3. Take snapshot

sudo rke2 etcd-snapshot save
sudo rke2 etcd-snapshot list

kubectl get configmap -n kube-system rke2-etcd-snapshots -o yaml

While validating confirm its empty or does not exist

kubectl create configmap -n kube-system k3s-etcd-snapshot-extra-metadata --from-literal=foo=bar

sudo rke2 etcd-snapshot save
sudo rke2 etcd-snapshot list

Check configmap
While reproducing all snapshots created with extra metadata as well as without extra metadata is listed. As part of validation snapshots created only with extra metadata is present in the configmap

kubectl get configmap -n kube-system k3s-etcd-snapshots -o yaml

Reproducing the issue

ubuntu@ip-172-31-16-62:~$ sudo rke2 etcd-snapshot save
INFO[0000] Saving etcd snapshot to /var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-16-62-1701474621 
{"level":"info","ts":"2023-12-01T23:50:20.952Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-16-62-1701474621.part"}
{"level":"info","ts":"2023-12-01T23:50:20.957Z","logger":"client","caller":"[email protected]/maintenance.go:211","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-12-01T23:50:20.961Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-12-01T23:50:21.187Z","logger":"client","caller":"[email protected]/maintenance.go:219","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-12-01T23:50:21.214Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"7.3 MB","took":"now"}
{"level":"info","ts":"2023-12-01T23:50:21.215Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-16-62-1701474621"}
INFO[0000] Reconciling ETCDSnapshotFile resources       
INFO[0000] Reconciliation of ETCDSnapshotFile resources complete 

ubuntu@ip-172-31-16-62:~$ kubectl get configmap -n kube-system rke2-etcd-snapshots -o yaml
apiVersion: v1
data:
  local-on-demand-ip-172-31-16-62-1701474616: '{"name":"on-demand-ip-172-31-16-62-1701474616","location":"file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-16-62-1701474616","nodeName":"ip-172-31-16-62","createdAt":"2023-12-01T23:50:16Z","size":7233568,"status":"successful","compressed":false}'
  local-on-demand-ip-172-31-16-62-1701474621: '{"name":"on-demand-ip-172-31-16-62-1701474621","location":"file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-16-62-1701474621","nodeName":"ip-172-31-16-62","createdAt":"2023-12-01T23:50:21Z","size":7266336,"status":"successful","compressed":false}'
kind: ConfigMap
metadata:
  creationTimestamp: "2023-12-01T23:50:15Z"
  name: rke2-etcd-snapshots
  namespace: kube-system
  resourceVersion: "1792"
  uid: 163e6b4b-dd27-4a02-9615-5d3b0bb187ea

ubuntu@ip-172-31-16-62:~$ kubectl create configmap -n kube-system rke2-etcd-snapshot-extra-metadata --from-literal=foo=bar 
configmap/rke2-etcd-snapshot-extra-metadata created

ubuntu@ip-172-31-16-62:~$ sudo rke2 etcd-snapshot save
INFO[0000] Saving etcd snapshot to /var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-16-62-1701474878 
{"level":"info","ts":"2023-12-01T23:54:38.263Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-16-62-1701474878.part"}
{"level":"info","ts":"2023-12-01T23:54:38.265Z","logger":"client","caller":"[email protected]/maintenance.go:211","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-12-01T23:54:38.266Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-12-01T23:54:38.409Z","logger":"client","caller":"[email protected]/maintenance.go:219","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-12-01T23:54:38.441Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"8.1 MB","took":"now"}
{"level":"info","ts":"2023-12-01T23:54:38.442Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-16-62-1701474878"}
INFO[0000] Saving snapshot metadata to /var/lib/rancher/rke2/server/db/.metadata/on-demand-ip-172-31-16-62-1701474878 
INFO[0000] Reconciling ETCDSnapshotFile resources       
INFO[0000] Reconciliation of ETCDSnapshotFile resources complete 

ubuntu@ip-172-31-16-62:~$ kubectl get configmap -n kube-system rke2-etcd-snapshots -o yaml
apiVersion: v1
data:
  local-on-demand-ip-172-31-16-62-1701474616: '{"name":"on-demand-ip-172-31-16-62-1701474616","location":"file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-16-62-1701474616","nodeName":"ip-172-31-16-62","createdAt":"2023-12-01T23:50:16Z","size":7233568,"status":"successful","compressed":false}'
  local-on-demand-ip-172-31-16-62-1701474621: '{"name":"on-demand-ip-172-31-16-62-1701474621","location":"file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-16-62-1701474621","nodeName":"ip-172-31-16-62","createdAt":"2023-12-01T23:50:21Z","size":7266336,"status":"successful","compressed":false}'
  local-on-demand-ip-172-31-16-62-1701474878: '{"name":"on-demand-ip-172-31-16-62-1701474878","location":"file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-16-62-1701474878","metadata":"eyJmb28iOiJiYXIifQ==","nodeName":"ip-172-31-16-62","createdAt":"2023-12-01T23:54:38Z","size":8073248,"status":"successful","compressed":false}'
kind: ConfigMap
metadata:
  creationTimestamp: "2023-12-01T23:50:15Z"
  name: rke2-etcd-snapshots
  namespace: kube-system
  resourceVersion: "2829"
  uid: 163e6b4b-dd27-4a02-9615-5d3b0bb187ea

Validating the issue

$ sudo rke2 etcd-snapshot save
INFO[0000] Saving etcd snapshot to /var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-28-49-1701467022 
{"level":"info","ts":"2023-12-01T21:43:42.370Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-28-49-1701467022.part"}
{"level":"info","ts":"2023-12-01T21:43:42.374Z","logger":"client","caller":"[email protected]/maintenance.go:211","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-12-01T21:43:42.374Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-12-01T21:43:42.640Z","logger":"client","caller":"[email protected]/maintenance.go:219","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-12-01T21:43:42.660Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"7.0 MB","took":"now"}
{"level":"info","ts":"2023-12-01T21:43:42.660Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-28-49-1701467022"}
INFO[0000] Reconciling ETCDSnapshotFile resources       
INFO[0000] Reconciliation of ETCDSnapshotFile resources complete

ubuntu@ip-172-31-28-49:~$ sudo rke2 etcd-snapshot list
Name                                 Location                                                                              Size    Created
on-demand-ip-172-31-28-49-1701467022 file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-28-49-1701467022 6955040 2023-12-01T21:43:42Z

configmap does not exist

ubuntu@ip-172-31-28-49:~$ kubectl get configmap -n kube-system rke2-etcd-snapshots -o yaml
Error from server (NotFound): configmaps "rke2-etcd-snapshots" not found
ubuntu@ip-172-31-28-49:~$ kubectl get configmap -n kube-system k3s-etcd-snapshot-extra-metadata -o yaml
Error from server (NotFound): configmaps "k3s-etcd-snapshot-extra-metadata" not found

Create configmap

$ kubectl create configmap -n kube-system rke2-etcd-snapshot-extra-metadata --from-literal=foo=bar

Take another snapshot

$ sudo rke2 etcd-snapshot save
INFO[0000] Saving etcd snapshot to /var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-28-49-1701472607 
{"level":"info","ts":"2023-12-01T23:16:47.185Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-28-49-1701472607.part"}
{"level":"info","ts":"2023-12-01T23:16:47.189Z","logger":"client","caller":"[email protected]/maintenance.go:211","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-12-01T23:16:47.189Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-12-01T23:16:47.471Z","logger":"client","caller":"[email protected]/maintenance.go:219","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-12-01T23:16:47.499Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"10 MB","took":"now"}
{"level":"info","ts":"2023-12-01T23:16:47.499Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-28-49-1701472607"}
INFO[0000] Saving snapshot metadata to /var/lib/rancher/rke2/server/db/.metadata/on-demand-ip-172-31-28-49-1701472607 
INFO[0000] Reconciling ETCDSnapshotFile resources       
INFO[0000] Reconciliation of ETCDSnapshotFile resources complete 

Validate both snapshots are listed

ubuntu@ip-172-31-28-49:~$ sudo rke2 etcd-snapshot list
Name                                 Location                                                                              Size     Created
on-demand-ip-172-31-28-49-1701467022 file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-28-49-1701467022 6955040  2023-12-01T21:43:42Z
on-demand-ip-172-31-28-49-1701472607 file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-28-49-1701472607 10383392 2023-12-01T23:16:47Z

configmap contains snapshots created with extra metadata only

ubuntu@ip-172-31-28-49:~$ kubectl get configmap -n kube-system rke2-etcd-snapshots -o yaml
apiVersion: v1
data:
  local-on-demand-ip-172-31-28-49-1701472607: '{"name":"on-demand-ip-172-31-28-49-1701472607","location":"file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-28-49-1701472607","metadata":"eyJmb28iOiJiYXIifQ==","nodeName":"ip-172-31-28-49","createdAt":"2023-12-01T23:16:47Z","size":10383392,"status":"successful","compressed":false}'
kind: ConfigMap
metadata:
  creationTimestamp: "2023-12-01T23:16:47Z"
  name: rke2-etcd-snapshots
  namespace: kube-system
  resourceVersion: "30613"
  uid: 21936e60-64fd-4264-9ba0-f2468fdd17ff