[Release-1.27] - [BUG] rke2 initial auto ETCD snapshot created with 0B data, unable to restore to snapshot

[brandond]

Backport fix for [BUG] rke2 initial auto ETCD snapshot created with 0B data, unable to restore to snapshot

• [BUG] rke2 initial auto ETCD snapshot created with 0B data, unable to restore to snapshot #5008

[endawkins]

Validated on 1.27 with 31e6aaa / 1.27

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

Linux ip-172-31-10-3 5.14.21-150400.22-default #1 SMP PREEMPT_DYNAMIC Wed May 11 06:57:18 UTC 2022 (49db222) x86_64 x86_64 x86_64 GNU/Linux
NAME="SLES"
VERSION="15-SP4"
VERSION_ID="15.4"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP4"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15:sp4"
DOCUMENTATION_URL="https://documentation.suse.com/"

Cluster Configuration:

1 server
1 agent

Config.yaml:

N/A

Additional files

N/A

Testing Steps

1. Copy config.yaml

$ sudo mkdir -p /etc/rancher/rke2 && sudo cp config.yaml /etc/rancher/rke2

2. Install RKE2
3. Run the following commands:

$ sudo rke2 etcd-snapshot save
$ sudo rke2 etcd-snapshot list - confirm snapshot is listed
$ kubectl get configmap -n kube-system rke2-etcd-snapshots -o yaml - confirm empty or does not exist
$ kubectl create configmap -n kube-system rke2-etcd-snapshot-extra-metadata --from-literal=foo=bar 
$ sudo rke2 etcd-snapshot save
$ sudo rke2 etcd-snapshot list - confirm both snapshots are listed
$ kubectl get configmap -n kube-system rke2-etcd-snapshots -o yaml - confirm only second snapshot is listed

Replication Results:

• rke2 version used for replication:

1. rke2 -v

rke2 version v1.27.7+rke2r2 (b0106a62b46c557e73512a02b2e4348e30dac7d7)
go version go1.20.10 X:boringcrypto

2. sudo rke2 etcd-snapshot save

INFO[0000] Saving etcd snapshot to /var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-8-128.us-east-2.compute.internal-1701514807
{"level":"info","ts":"2023-12-02T11:00:07.484456Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-8-128.us-east-2.compute.internal-1701514807.part"}
{"level":"info","ts":"2023-12-02T11:00:07.490422Z","logger":"client","caller":"[email protected]/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-12-02T11:00:07.490486Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-12-02T11:00:07.681738Z","logger":"client","caller":"[email protected]/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-12-02T11:00:07.908011Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"11 MB","took":"now"}
{"level":"info","ts":"2023-12-02T11:00:07.908106Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-8-128.us-east-2.compute.internal-1701514807"}
INFO[0000] Reconciling ETCDSnapshotFile resources
INFO[0000] Reconciliation of ETCDSnapshotFile resources complete

3. sudo rke2 etcd-snapshot list

Name                                                            Location                                                                                                         Size     Created
on-demand-ip-172-31-8-128.us-east-2.compute.internal-1701514807 file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-8-128.us-east-2.compute.internal-1701514807 11276320 2023-12-02T11:00:07Z

4. kubectl get configmap -n kube-system rke2-etcd-snapshots -o yaml

apiVersion: v1
data:
  local-on-demand-ip-172-31-8-128.us-east-2.compute.internal-1701514807: '{"name":"on-demand-ip-172-31-8-128.us-east-2.compute.internal-1701514807","location":"file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-8-128.us-east-2.compute.internal-1701514807","nodeName":"ip-172-31-8-128.us-east-2.compute.internal","createdAt":"2023-12-02T11:00:07Z","size":11276320,"status":"successful","compressed":false}'
kind: ConfigMap
metadata:
  creationTimestamp: "2023-12-02T11:00:07Z"
  name: rke2-etcd-snapshots
  namespace: kube-system
  resourceVersion: "4402"
  uid: d5d18cba-2bc1-45b5-bb26-d973dab4ae76

5. kubectl create configmap -n kube-system rke2-etcd-snapshot-extra-metadata --from-literal=foo=bar

configmap/rke2-etcd-snapshot-extra-metadata created

6. sudo rke2 etcd-snapshot save

INFO[0000] Saving etcd snapshot to /var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-8-128.us-east-2.compute.internal-1701514898
{"level":"info","ts":"2023-12-02T11:01:37.760737Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-8-128.us-east-2.compute.internal-1701514898.part"}
{"level":"info","ts":"2023-12-02T11:01:37.765969Z","logger":"client","caller":"[email protected]/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-12-02T11:01:37.76604Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-12-02T11:01:37.97698Z","logger":"client","caller":"[email protected]/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-12-02T11:01:38.209739Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"11 MB","took":"now"}
{"level":"info","ts":"2023-12-02T11:01:38.209851Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-8-128.us-east-2.compute.internal-1701514898"}
INFO[0000] Saving snapshot metadata to /var/lib/rancher/rke2/server/db/.metadata/on-demand-ip-172-31-8-128.us-east-2.compute.internal-1701514898
INFO[0000] Reconciling ETCDSnapshotFile resources
INFO[0000] Reconciliation of ETCDSnapshotFile resources complete

7. sudo rke2 etcd-snapshot list

Name                                                            Location                                                                                                         Size     Created
on-demand-ip-172-31-8-128.us-east-2.compute.internal-1701514807 file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-8-128.us-east-2.compute.internal-1701514807 11276320 2023-12-02T11:00:07Z
on-demand-ip-172-31-8-128.us-east-2.compute.internal-1701514898 file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-8-128.us-east-2.compute.internal-1701514898 11276320 2023-12-02T11:01:38Z

8. kubectl get configmap -n kube-system rke2-etcd-snapshots -o yaml

apiVersion: v1
data:
  local-on-demand-ip-172-31-8-128.us-east-2.compute.internal-1701514807: '{"name":"on-demand-ip-172-31-8-128.us-east-2.compute.internal-1701514807","location":"file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-8-128.us-east-2.compute.internal-1701514807","nodeName":"ip-172-31-8-128.us-east-2.compute.internal","createdAt":"2023-12-02T11:00:07Z","size":11276320,"status":"successful","compressed":false}'
  local-on-demand-ip-172-31-8-128.us-east-2.compute.internal-1701514898: '{"name":"on-demand-ip-172-31-8-128.us-east-2.compute.internal-1701514898","location":"file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-8-128.us-east-2.compute.internal-1701514898","metadata":"eyJmb28iOiJiYXIifQ==","nodeName":"ip-172-31-8-128.us-east-2.compute.internal","createdAt":"2023-12-02T11:01:38Z","size":11276320,"status":"successful","compressed":false}'
kind: ConfigMap
metadata:
  creationTimestamp: "2023-12-02T11:00:07Z"
  name: rke2-etcd-snapshots
  namespace: kube-system
  resourceVersion: "4782"
  uid: d5d18cba-2bc1-45b5-bb26-d973dab4ae76

Validation Results:

• rke2 version used for validation:

1. rke2 -v

rke2 version v1.27.8-rc1+rke2r1 (31e6aaa64fc5bf6c416737443239b3f58515b15e)
go version go1.20.11 X:boringcrypto

2. sudo rke2 etcd-snapshot save

INFO[0000] Saving etcd snapshot to /var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-15-129.us-east-2.compute.internal-1701515013
{"level":"info","ts":"2023-12-02T11:03:32.679656Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-15-129.us-east-2.compute.internal-1701515013.part"}
{"level":"info","ts":"2023-12-02T11:03:32.799369Z","logger":"client","caller":"[email protected]/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-12-02T11:03:32.799408Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-12-02T11:03:32.945024Z","logger":"client","caller":"[email protected]/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-12-02T11:03:33.910593Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"11 MB","took":"1 second ago"}
{"level":"info","ts":"2023-12-02T11:03:33.910696Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-15-129.us-east-2.compute.internal-1701515013"}
INFO[0001] Reconciling ETCDSnapshotFile resources
INFO[0001] Reconciliation of ETCDSnapshotFile resources complete

3. sudo rke2 etcd-snapshot list

Name                                                             Location                                                                                                          Size     Created
on-demand-ip-172-31-15-129.us-east-2.compute.internal-1701515013 file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-15-129.us-east-2.compute.internal-1701515013 11190304 2023-12-02T11:03:33Z

4. kubectl get configmap -n kube-system rke2-etcd-snapshots -o yaml

Error from server (NotFound): configmaps "rke2-etcd-snapshots" not found

5. kubectl create configmap -n kube-system rke2-etcd-snapshot-extra-metadata --from-literal=foo=bar

configmap/rke2-etcd-snapshot-extra-metadata created

6. sudo rke2 etcd-snapshot save

INFO[0000] Saving etcd snapshot to /var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-15-129.us-east-2.compute.internal-1701515079
{"level":"info","ts":"2023-12-02T11:04:38.933442Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-15-129.us-east-2.compute.internal-1701515079.part"}
{"level":"info","ts":"2023-12-02T11:04:38.941928Z","logger":"client","caller":"[email protected]/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-12-02T11:04:38.942066Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-12-02T11:04:39.114478Z","logger":"client","caller":"[email protected]/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-12-02T11:04:39.150715Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"11 MB","took":"now"}
{"level":"info","ts":"2023-12-02T11:04:39.150821Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-15-129.us-east-2.compute.internal-1701515079"}
INFO[0000] Saving snapshot metadata to /var/lib/rancher/rke2/server/db/.metadata/on-demand-ip-172-31-15-129.us-east-2.compute.internal-1701515079
INFO[0000] Reconciling ETCDSnapshotFile resources
INFO[0000] Reconciliation of ETCDSnapshotFile resources complete

7. sudo rke2 etcd-snapshot list

Name                                                             Location                                                                                                          Size     Created
on-demand-ip-172-31-15-129.us-east-2.compute.internal-1701515013 file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-15-129.us-east-2.compute.internal-1701515013 11190304 2023-12-02T11:03:33Z
on-demand-ip-172-31-15-129.us-east-2.compute.internal-1701515079 file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-15-129.us-east-2.compute.internal-1701515079 11190304 2023-12-02T11:04:39Z

8. kubectl get configmap -n kube-system rke2-etcd-snapshots -o yaml

apiVersion: v1
data:
  local-on-demand-ip-172-31-15-129.us-east-2.compute.internal-1701515079: '{"name":"on-demand-ip-172-31-15-129.us-east-2.compute.internal-1701515079","location":"file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-15-129.us-east-2.compute.internal-1701515079","metadata":"eyJmb28iOiJiYXIifQ==","nodeName":"ip-172-31-15-129.us-east-2.compute.internal","createdAt":"2023-12-02T11:04:39Z","size":11190304,"status":"successful","compressed":false}'
kind: ConfigMap
metadata:
  creationTimestamp: "2023-12-02T11:04:39Z"
  name: rke2-etcd-snapshots
  namespace: kube-system
  resourceVersion: "5416"
  uid: a61d47f5-6790-4524-b40d-1eb860065c35

Additional context / logs:

N/A