Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Release-1.25] - [BUG] rke2-etcd-snapshots configmap grows too large when snapshot retention is high #4878

Closed
brandond opened this issue Oct 12, 2023 · 2 comments
Closed

[Release-1.25] - [BUG] rke2-etcd-snapshots configmap grows too large when snapshot retention is high #4878

brandond opened this issue Oct 12, 2023 · 2 comments
Assignees
@brandond @aganesh-suse
Milestone
v1.25.15+rke2r1

Comments

@brandond
Copy link
Member

Backport fix for [BUG] rke2-etcd-snapshots configmap grows too large when snapshot retention is high
@brandond brandond self-assigned this Oct 12, 2023
@brandond brandond mentioned this issue Oct 13, 2023
Merged
@caroline-suse-rancher caroline-suse-rancher added this to the v1.25.15+rke2r1 milestone Oct 16, 2023
@aganesh-suse aganesh-suse self-assigned this Oct 19, 2023
@aganesh-suse
Copy link

Validated on release-1.25 branch with commit 45f7f19

Environment Details

Infrastructure

  • Cloud
  • Hosted

Node(s) CPU architecture, OS, and Version:

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"

$ uname -m
x86_64

Cluster Configuration:

HA : 1 server / 1 agent

Config.yaml:

token: secret
write-kubeconfig-mode: "0644"
node-external-ip: 1.1.1.1
etcd-snapshot-retention: 5
etcd-snapshot-schedule-cron: "* * * * *"
etcd-s3: true
etcd-s3-access-key: xxxx
etcd-s3-secret-key: xxxx
etcd-s3-bucket: xxxx
etcd-s3-folder: xxxx/rke2-1
etcd-s3-region: xxxx

Additional files

custom.yaml content:
https://gist.githubusercontent.com/aganesh-suse/b8f776fdcc4598ca25cd386ac5208c51/raw/2e8fb77643f7bd9e75e8a94e26ff8965e23ccd5c/rke2-etcd-snapshot-extra-metadata.yaml

Testing Steps

  1. Copy config.yaml
$ sudo mkdir -p /etc/rancher/rke2 && sudo cp config.yaml /etc/rancher/rke2
  1. Install RKE2
curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_COMMIT='45f7f1922346518ffab0db448b98a9ebd44b91c2' INSTALL_RKE2_TYPE='server' INSTALL_RKE2_METHOD=tar sh -
  1. Start the RKE2 service
$ sudo systemctl enable --now rke2-server
or 
$ sudo systemctl enable --now rke2-agent
  1. Verify Cluster Status:
kubectl get nodes -o wide
kubectl get pods -A
  1. Apply rke2 snapshot metadata yaml file. Save the etcd-snapshot file 5 times.
$ kubectl apply -f custom.yaml
or
$ kubectl apply -f https://gist.githubusercontent.com/aganesh-suse/b8f776fdcc4598ca25cd386ac5208c51/raw/2e8fb77643f7bd9e75e8a94e26ff8965e23ccd5c/rke2-etcd-snapshot-extra-metadata.yaml
$ for (( I=0; I < 5; I++ )); do sudo rke2 etcd-snapshot save ; done
  1. Verify etcd snapshot operations:
sudo rke2 etcd-snapshot save
sudo rke2 etcd-snapshot list
sudo rke2 etcd-snapshot prune --snapshot-retention <retention_count>
sudo rke2 etcd-snapshot delete <snapshot_name>

Replication Results:

  • rke2 version used for replication:
rke2 -v
rke2 version v1.25.14+rke2r1 (36d7417e024e8dad34ebbf94b210ab3dd0f52cd7)
go version go1.20.8 X:boringcrypto

$ sudo rke2 etcd-snapshot save
time="2023-10-21T02:31:10Z" level=warning msg="Unknown flag --token found in config.yaml, skipping\n"
time="2023-10-21T02:31:10Z" level=warning msg="Unknown flag --etcd-snapshot-retention found in config.yaml, skipping\n"
time="2023-10-21T02:31:10Z" level=warning msg="Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping\n"
time="2023-10-21T02:31:10Z" level=warning msg="Unknown flag --write-kubeconfig-mode found in config.yaml, skipping\n"
time="2023-10-21T02:31:10Z" level=warning msg="Unknown flag --node-external-ip found in config.yaml, skipping\n"
time="2023-10-21T02:31:10Z" level=info msg="Managed etcd cluster bootstrap already complete and initialized"
time="2023-10-21T02:31:10Z" level=info msg="Applying CRD helmcharts.helm.cattle.io"
time="2023-10-21T02:31:10Z" level=info msg="Applying CRD helmchartconfigs.helm.cattle.io"
time="2023-10-21T02:31:10Z" level=info msg="Applying CRD addons.k3s.cattle.io"
time="2023-10-21T02:31:10Z" level=info msg="Creating rke2-supervisor event broadcaster"
time="2023-10-21T02:31:10Z" level=info msg="Saving etcd snapshot to /var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-20-10-1697855470"
{"level":"info","ts":"2023-10-21T02:31:10.460Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-20-10-1697855470.part"}
{"level":"info","ts":"2023-10-21T02:31:10.464Z","logger":"client","caller":"[email protected]/maintenance.go:211","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-10-21T02:31:10.464Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-10-21T02:31:10.921Z","logger":"client","caller":"[email protected]/maintenance.go:219","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-10-21T02:31:10.963Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"20 MB","took":"now"}
{"level":"info","ts":"2023-10-21T02:31:10.963Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-172-31-20-10-1697855470"}
time="2023-10-21T02:31:11Z" level=fatal msg="failed to save local snapshot data to configmap: ConfigMap \"rke2-etcd-snapshots\" is invalid: []: Too long: must have at most 1048576 bytes"

$ sudo rke2 etcd-snapshot list
Name                                     Size     Created
etcd-snapshot-ip-x-x-x-x-1697839142 15298592 2023-10-20T21:59:04Z
etcd-snapshot-ip-x-x-x-x-1697839442 18112544 2023-10-20T22:04:04Z
etcd-snapshot-ip-x-x-x-x-1697839504 22462496 2023-10-20T22:05:06Z
on-demand-ip-x-x-x-x-1697839128     10944544 2023-10-20T21:58:49Z
on-demand-ip-x-x-x-x-1697839139     11886624 2023-10-20T21:59:01Z
etcd-snapshot-ip-x-x-x-x-1697839020 8712224  2023-10-20T21:57:01Z
etcd-snapshot-ip-x-x-x-x-1697839081 9256992  2023-10-20T21:58:03Z

$ kubectl get cm -n kube-system | grep etcd
rke2-etcd                                        0      27m
rke2-etcd-snapshot-extra-metadata                1      17m
rke2-etcd-snapshots                              16     27m

Validation Results:

  • rke2 version used for validation:
$ rke2 -v
rke2 version v1.25.15-rc2+rke2r1 (45f7f1922346518ffab0db448b98a9ebd44b91c2)
go version go1.20.10 X:boringcrypto

ETCD Snapshot Operation Results:

$ kubectl get cm -n kube-system | grep etcd
rke2-etcd                                        0      24m
rke2-etcd-snapshot-extra-metadata                1      17m
rke2-etcd-snapshots                              8      23m

$ sudo ls -lrt /var/lib/rancher/rke2/server/db/snapshots 
total 133604
-rw------- 1 root root  8114208 Oct 21 02:27 etcd-snapshot-ip-x-x-x-x-1697855221
-rw------- 1 root root  8343584 Oct 21 02:28 etcd-snapshot-ip-x-x-x-x-1697855284
-rw------- 1 root root  8593440 Oct 21 02:29 etcd-snapshot-ip-x-x-x-x-1697855344
-rw------- 1 root root  9760800 Oct 21 02:30 etcd-snapshot-ip-x-x-x-x-1697855402
-rw------- 1 root root 11366432 Oct 21 02:30 on-demand-ip-x-x-x-x-1697855438
-rw------- 1 root root 13619232 Oct 21 02:30 on-demand-ip-x-x-x-x-1697855450
-rw------- 1 root root 16064544 Oct 21 02:31 etcd-snapshot-ip-x-x-x-x-1697855461
-rw------- 1 root root 16064544 Oct 21 02:31 on-demand-ip-x-x-x-x-1697855461
-rw------- 1 root root 21495840 Oct 21 02:31 on-demand-ip-x-x-x-x-1697855472
-rw------- 1 root root 23347232 Oct 21 02:31 on-demand-ip-x-x-x-x-1697855483

$ sudo ls -lrt /var/lib/rancher/rke2/server/db/snapshots | grep -v total | wc -l 
10

$ sudo rke2 etcd-snapshot list 2>/dev/null 
Name                                      Location                                                                                           Size     Created
etcd-snapshot-ip-x-x-x-x-1697855221 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1697855221         8114208  2023-10-21T02:27:01Z
etcd-snapshot-ip-x-x-x-x-1697855221 s3://xxxx/xxxx/rke2-2/etcd-snapshot-ip-x-x-x-x-1697855221 8114208  2023-10-21T02:27:01Z
etcd-snapshot-ip-x-x-x-x-1697855284 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1697855284         8343584  2023-10-21T02:28:04Z
etcd-snapshot-ip-x-x-x-x-1697855284 s3://xxxx/xxxx/rke2-2/etcd-snapshot-ip-x-x-x-x-1697855284 8343584  2023-10-21T02:28:04Z
etcd-snapshot-ip-x-x-x-x-1697855344 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1697855344         8593440  2023-10-21T02:29:04Z
etcd-snapshot-ip-x-x-x-x-1697855344 s3://xxxx/xxxx/rke2-2/etcd-snapshot-ip-x-x-x-x-1697855344 8593440  2023-10-21T02:29:04Z
etcd-snapshot-ip-x-x-x-x-1697855402 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1697855402         9760800  2023-10-21T02:30:02Z
etcd-snapshot-ip-x-x-x-x-1697855402 s3://xxxx/xxxx/rke2-2/etcd-snapshot-ip-x-x-x-x-1697855402 9760800  2023-10-21T02:30:02Z
on-demand-ip-x-x-x-x-1697855438     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697855438             11366432 2023-10-21T02:30:38Z
on-demand-ip-x-x-x-x-1697855438     s3://xxxx/xxxx/rke2-2/on-demand-ip-x-x-x-x-1697855438     11366432 2023-10-21T02:30:38Z
on-demand-ip-x-x-x-x-1697855450     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697855450             13619232 2023-10-21T02:30:50Z
on-demand-ip-x-x-x-x-1697855450     s3://xxxx/xxxx/rke2-2/on-demand-ip-x-x-x-x-1697855450     13619232 2023-10-21T02:30:50Z
etcd-snapshot-ip-x-x-x-x-1697855461 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1697855461         16064544 2023-10-21T02:31:01Z
on-demand-ip-x-x-x-x-1697855461     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697855461             16064544 2023-10-21T02:31:01Z
etcd-snapshot-ip-x-x-x-x-1697855461 s3://xxxx/xxxx/rke2-2/etcd-snapshot-ip-x-x-x-x-1697855461 16064544 2023-10-21T02:31:01Z
on-demand-ip-x-x-x-x-1697855461     s3://xxxx/xxxx/rke2-2/on-demand-ip-x-x-x-x-1697855461     16064544 2023-10-21T02:31:01Z
on-demand-ip-x-x-x-x-1697855472     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697855472             21495840 2023-10-21T02:31:12Z
on-demand-ip-x-x-x-x-1697855472     s3://xxxx/xxxx/rke2-2/on-demand-ip-x-x-x-x-1697855472     21495840 2023-10-21T02:31:12Z
on-demand-ip-x-x-x-x-1697855483     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697855483             23347232 2023-10-21T02:31:23Z
on-demand-ip-x-x-x-x-1697855483     s3://xxxx/xxxx/rke2-2/on-demand-ip-x-x-x-x-1697855483     23347232 2023-10-21T02:31:23Z

$ sudo rke2 etcd-snapshot save 
time="2023-10-21T02:31:12Z" level=warning msg="Unknown flag --token found in config.yaml, skipping\n"
time="2023-10-21T02:31:12Z" level=warning msg="Unknown flag --etcd-snapshot-retention found in config.yaml, skipping\n"
time="2023-10-21T02:31:12Z" level=warning msg="Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping\n"
time="2023-10-21T02:31:12Z" level=warning msg="Unknown flag --write-kubeconfig-mode found in config.yaml, skipping\n"
time="2023-10-21T02:31:12Z" level=warning msg="Unknown flag --node-external-ip found in config.yaml, skipping\n"
time="2023-10-21T02:31:12Z" level=info msg="Saving etcd snapshot to /var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697855472"
{"level":"info","ts":"2023-10-21T02:31:12.498Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697855472.part"}
{"level":"info","ts":"2023-10-21T02:31:12.501Z","logger":"client","caller":"[email protected]/maintenance.go:211","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-10-21T02:31:12.501Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-10-21T02:31:12.988Z","logger":"client","caller":"[email protected]/maintenance.go:219","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-10-21T02:31:13.046Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"22 MB","took":"now"}
{"level":"info","ts":"2023-10-21T02:31:13.046Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697855472"}
time="2023-10-21T02:31:13Z" level=info msg="Saving snapshot metadata to /var/lib/rancher/rke2/server/db/.metadata/on-demand-ip-x-x-x-x-1697855472"
time="2023-10-21T02:31:13Z" level=info msg="Checking if S3 bucket sonobuoy-results exists"
time="2023-10-21T02:31:13Z" level=info msg="S3 bucket sonobuoy-results exists"
time="2023-10-21T02:31:13Z" level=info msg="Saving etcd snapshot on-demand-ip-x-x-x-x-1697855472 to S3"
time="2023-10-21T02:31:13Z" level=info msg="Uploading snapshot to s3://sonobuoy-results//var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697855472"
time="2023-10-21T02:31:13Z" level=info msg="Uploaded snapshot metadata s3://sonobuoy-results//var/lib/rancher/rke2/server/db/.metadata/on-demand-ip-x-x-x-x-1697855472"
time="2023-10-21T02:31:13Z" level=info msg="S3 upload complete for on-demand-ip-x-x-x-x-1697855472"
time="2023-10-21T02:31:13Z" level=info msg="Reconciling ETCDSnapshotFile resources"
time="2023-10-21T02:31:14Z" level=info msg="Reconciliation of ETCDSnapshotFile resources complete"

$ sudo rke2 etcd-snapshot prune --snapshot-retention 3
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --etcd-snapshot-retention found in config.yaml, skipping
WARN[0000] Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
INFO[0000] Applying snapshot retention=3 to local snapshots with prefix on-demand in /var/lib/rancher/rke2/server/db/snapshots
INFO[0000] Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697847673
INFO[0000] Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697847669
INFO[0000] Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697842551
INFO[0000] Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697839356
INFO[0000] Checking if S3 bucket xxxx exists
INFO[0000] S3 bucket xxxx exists
INFO[0000] Applying snapshot retention=3 to snapshots stored in s3://xxxx/arch-rke2snap/version-setup/rke2-2/on-demand
INFO[0000] Removing S3 snapshot: s3://xxxx/arch-rke2snap/version-setup/rke2-2/on-demand-ip-x-x-x-x-1697847673
INFO[0000] Removing S3 snapshot: s3://xxxx/arch-rke2snap/version-setup/rke2-2/on-demand-ip-x-x-x-x-1697847669
INFO[0000] Removing S3 snapshot: s3://xxxx/arch-rke2snap/version-setup/rke2-2/on-demand-ip-x-x-x-x-1697842551
INFO[0000] Removing S3 snapshot: s3://xxxx/arch-rke2snap/version-setup/rke2-2/on-demand-ip-x-x-x-x-1697839356
INFO[0000] Reconciling ETCDSnapshotFile resources
INFO[0001] Deleting ETCDSnapshotFile for on-demand-ip-x-x-x-x-1697839356
INFO[0001] Deleting ETCDSnapshotFile for on-demand-ip-x-x-x-x-1697842551
INFO[0001] Deleting ETCDSnapshotFile for on-demand-ip-x-x-x-x-1697847669
INFO[0001] Deleting ETCDSnapshotFile for on-demand-ip-x-x-x-x-1697847673
INFO[0001] Deleting ETCDSnapshotFile for on-demand-ip-x-x-x-x-1697839356
INFO[0001] Deleting ETCDSnapshotFile for on-demand-ip-x-x-x-x-1697842551
INFO[0001] Deleting ETCDSnapshotFile for on-demand-ip-x-x-x-x-1697847669
INFO[0001] Deleting ETCDSnapshotFile for on-demand-ip-x-x-x-x-1697847673
INFO[0001] Reconciliation of ETCDSnapshotFile resources complete

sudo rke2 etcd-snapshot delete on-demand-ip-x-x-x-x-1697839350
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --etcd-snapshot-retention found in config.yaml, skipping
WARN[0000] Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
INFO[0000] Checking if S3 bucket xxxx exists
INFO[0000] S3 bucket xxxx exists
INFO[0000] Snapshot on-demand-ip-x-x-x-x-1697839350 deleted locally
INFO[0000] Snapshot on-demand-ip-x-x-x-x-1697839350 deleted from S3
INFO[0000] Reconciling ETCDSnapshotFile resources
INFO[0000] Deleting ETCDSnapshotFile for on-demand-ip-x-x-x-x-1697839350
INFO[0000] Deleting ETCDSnapshotFile for on-demand-ip-x-x-x-x-1697839350
INFO[0000] Reconciliation of ETCDSnapshotFile resources complete

$ /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get etcdsnapshotfile
NAME                                                     SNAPSHOTNAME                                NODE               LOCATION                                                                                             SIZE       CREATIONTIME
local-etcd-snapshot-ip-1x-x-x-x-1697855221-204c96   etcd-snapshot-ip-1x-x-x-x-1697855221   ip-1x-x-x-x   file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-1x-x-x-x-1697855221           8114208    2023-10-21T02:27:01Z
local-etcd-snapshot-ip-1x-x-x-x-1697855284-282a00   etcd-snapshot-ip-1x-x-x-x-1697855284   ip-1x-x-x-x   file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-1x-x-x-x-1697855284           8343584    2023-10-21T02:28:04Z
local-etcd-snapshot-ip-1x-x-x-x-1697855344-0c86aa   etcd-snapshot-ip-1x-x-x-x-1697855344   ip-1x-x-x-x   file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-1x-x-x-x-1697855344           8593440    2023-10-21T02:29:04Z
local-etcd-snapshot-ip-1x-x-x-x-1697855402-24f097   etcd-snapshot-ip-1x-x-x-x-1697855402   ip-1x-x-x-x   file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-1x-x-x-x-1697855402           9760800    2023-10-21T02:30:02Z
local-etcd-snapshot-ip-1x-x-x-x-1697855461-9742c3   etcd-snapshot-ip-1x-x-x-x-1697855461   ip-1x-x-x-x   file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-1x-x-x-x-1697855461           16064544   2023-10-21T02:31:01Z
local-on-demand-ip-1x-x-x-x-1697855438-25147a       on-demand-ip-1x-x-x-x-1697855438       ip-1x-x-x-x   file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-1x-x-x-x-1697855438               11366432   2023-10-21T02:30:38Z
local-on-demand-ip-1x-x-x-x-1697855450-a1f6b5       on-demand-ip-1x-x-x-x-1697855450       ip-1x-x-x-x   file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-1x-x-x-x-1697855450               13619232   2023-10-21T02:30:50Z
local-on-demand-ip-1x-x-x-x-1697855461-77eaed       on-demand-ip-1x-x-x-x-1697855461       ip-1x-x-x-x   file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-1x-x-x-x-1697855461               16064544   2023-10-21T02:31:01Z
local-on-demand-ip-1x-x-x-x-1697855472-ec1631       on-demand-ip-1x-x-x-x-1697855472       ip-1x-x-x-x   file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-1x-x-x-x-1697855472               21495840   2023-10-21T02:31:12Z
local-on-demand-ip-1x-x-x-x-1697855483-f92291       on-demand-ip-1x-x-x-x-1697855483       ip-1x-x-x-x   file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-1x-x-x-x-1697855483               23347232   2023-10-21T02:31:23Z
s3-etcd-snapshot-ip-1x-x-x-x-1697855221-0a77cf      etcd-snapshot-ip-1x-x-x-x-1697855221   ip-1x-x-x-x   s3://xxxx/xxxx/rke2-2/etcd-snapshot-ip-1x-x-x-x-1697855221   8114208    2023-10-21T02:27:01Z
s3-etcd-snapshot-ip-1x-x-x-x-1697855284-96b9bf      etcd-snapshot-ip-1x-x-x-x-1697855284   ip-1x-x-x-x   s3://xxxx/xxxx/rke2-2/etcd-snapshot-ip-1x-x-x-x-1697855284   8343584    2023-10-21T02:28:04Z
s3-etcd-snapshot-ip-1x-x-x-x-1697855344-d07761      etcd-snapshot-ip-1x-x-x-x-1697855344   ip-1x-x-x-x   s3://xxxx/xxxx/rke2-2/etcd-snapshot-ip-1x-x-x-x-1697855344   8593440    2023-10-21T02:29:04Z
s3-etcd-snapshot-ip-1x-x-x-x-1697855402-627d84      etcd-snapshot-ip-1x-x-x-x-1697855402   ip-1x-x-x-x   s3://xxxx/xxxx/rke2-2/etcd-snapshot-ip-1x-x-x-x-1697855402   9760800    2023-10-21T02:30:02Z
s3-etcd-snapshot-ip-1x-x-x-x-1697855461-0d3689      etcd-snapshot-ip-1x-x-x-x-1697855461   ip-1x-x-x-x   s3://xxxx/xxxx/rke2-2/etcd-snapshot-ip-1x-x-x-x-1697855461   16064544   2023-10-21T02:31:01Z
s3-on-demand-ip-1x-x-x-x-1697855438-451244          on-demand-ip-1x-x-x-x-1697855438       ip-1x-x-x-x   s3://xxxx/xxxx/rke2-2/on-demand-ip-1x-x-x-x-1697855438       11366432   2023-10-21T02:30:38Z
s3-on-demand-ip-1x-x-x-x-1697855450-2ad37b          on-demand-ip-1x-x-x-x-1697855450       ip-1x-x-x-x   s3://xxxx/xxxx/rke2-2/on-demand-ip-1x-x-x-x-1697855450       13619232   2023-10-21T02:30:50Z
s3-on-demand-ip-1x-x-x-x-1697855461-15e1de          on-demand-ip-1x-x-x-x-1697855461       ip-1x-x-x-x   s3://xxxx/xxxx/rke2-2/on-demand-ip-1x-x-x-x-1697855461       16064544   2023-10-21T02:31:01Z
s3-on-demand-ip-1x-x-x-x-1697855472-1647a1          on-demand-ip-1x-x-x-x-1697855472       ip-1x-x-x-x   s3://xxxx/xxxx/rke2-2/on-demand-ip-1x-x-x-x-1697855472       21495840   2023-10-21T02:31:12Z
s3-on-demand-ip-1x-x-x-x-1697855483-afc34a          on-demand-ip-1x-x-x-x-1697855483       ip-1x-x-x-x   s3://xxxx/xxxx/rke2-2/on-demand-ip-1x-x-x-x-1697855483       23347232   2023-10-21T02:31:23Z

@sadhugit
Copy link

HI team &@aganesh-suse, it looks issue still exists. it only works once then after removing the excess snapshots from s3 as per the retention policy it again fails to take snapshots and failing with the same error cc: @brandond `

Nov 16 07:12:00 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:00Z" level=info msg="wake, now=2023-11-16T07:12:00Z"
Nov 16 07:12:00 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:00Z" level=info msg="run, now=2023-11-16T07:12:00Z, entry=1, next=2023-11-16T07:14:00Z"
Nov 16 07:12:04 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:04Z" level=info msg="Saving etcd snapshot to /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700118724"
Nov 16 07:12:04 testissue-pool1-697442e0-br5np rke2[92866]: {"level":"info","ts":"2023-11-16T07:12:04.174926Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700118724.part"}
Nov 16 07:12:04 testissue-pool1-697442e0-br5np rke2[92866]: {"level":"info","ts":"2023-11-16T07:12:04.185125Z","logger":"client","caller":"[email protected]/maintenance.go:212","msg":"opened snapshot stream; downloading"}
Nov 16 07:12:04 testissue-pool1-697442e0-br5np rke2[92866]: {"level":"info","ts":"2023-11-16T07:12:04.185231Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: {"level":"info","ts":"2023-11-16T07:12:05.095559Z","logger":"client","caller":"[email protected]/maintenance.go:220","msg":"completed snapshot read; closing"}
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: {"level":"info","ts":"2023-11-16T07:12:05.112166Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"24 MB","took":"now"}
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: {"level":"info","ts":"2023-11-16T07:12:05.112589Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700118724"}
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Applying local snapshot retention policy: retention: 10, snapshotPrefix: etcd-snapshot, directory: /var/lib/rancher/rke2/server/db/snapshots"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700109362"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700109483"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700109605"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700109721"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700109840"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700109963"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700110084"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700110201"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700110320"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700110440"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700110562"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700110680"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700110801"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700110920"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700111040"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700111162"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700111281"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700111401"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700111522"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700111641"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700111760"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700111884"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700112000"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700112121"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700112242"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700112361"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700112482"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700112604"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700112723"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700112844"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700112962"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700113080"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700113203"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700113321"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700113440"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700113561"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700113681"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700113800"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700113921"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700114041"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700114162"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700114284"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700114402"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700114522"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700114644"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700114762"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700114882"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700115004"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700115122"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700115243"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700115361"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700115482"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700115604"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700115723"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700115840"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Saving etcd snapshot etcd-snapshot-testissue-pool1-697442e0-br5np-1700118724 to S3"
Nov 16 07:12:05 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:05Z" level=info msg="Uploading snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700118724 to S3"
Nov 16 07:12:06 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:06Z" level=info msg="S3 upload complete for etcd-snapshot-testissue-pool1-697442e0-br5np-1700118724"
Nov 16 07:12:06 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:06Z" level=info msg="Applying snapshot retention policy to snapshots stored in S3: retention: 10, snapshotPrefix: test/etcd-snapshot"
Nov 16 07:12:06 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:06Z" level=info msg="Removing S3 snapshot: test/etcd-snapshot-testissue-pool1-697442e0-d99kc-1700115720"
Nov 16 07:12:06 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:06Z" level=info msg="Reconciling etcd snapshot data in rke2-etcd-snapshots ConfigMap"
Nov 16 07:12:06 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:12:06Z" level=info msg="Reconciliation of snapshot data in rke2-etcd-snapshots ConfigMap complete"
Nov 16 07:14:00 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:14:00Z" level=info msg="wake, now=2023-11-16T07:14:00Z"
Nov 16 07:14:00 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:14:00Z" level=info msg="run, now=2023-11-16T07:14:00Z, entry=1, next=2023-11-16T07:16:00Z"
Nov 16 07:14:02 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:14:02Z" level=info msg="Saving etcd snapshot to /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700118842"
Nov 16 07:14:02 testissue-pool1-697442e0-br5np rke2[92866]: {"level":"info","ts":"2023-11-16T07:14:02.568259Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700118842.part"}
Nov 16 07:14:02 testissue-pool1-697442e0-br5np rke2[92866]: {"level":"info","ts":"2023-11-16T07:14:02.574601Z","logger":"client","caller":"[email protected]/maintenance.go:212","msg":"opened snapshot stream; downloading"}
Nov 16 07:14:02 testissue-pool1-697442e0-br5np rke2[92866]: {"level":"info","ts":"2023-11-16T07:14:02.575167Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
Nov 16 07:14:03 testissue-pool1-697442e0-br5np rke2[92866]: {"level":"info","ts":"2023-11-16T07:14:03.149641Z","logger":"client","caller":"[email protected]/maintenance.go:220","msg":"completed snapshot read; closing"}
Nov 16 07:14:03 testissue-pool1-697442e0-br5np rke2[92866]: {"level":"info","ts":"2023-11-16T07:14:03.215276Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"30 MB","took":"now"}
Nov 16 07:14:03 testissue-pool1-697442e0-br5np rke2[92866]: {"level":"info","ts":"2023-11-16T07:14:03.215404Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700118842"}
Nov 16 07:14:03 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:14:03Z" level=info msg="Applying local snapshot retention policy: retention: 10, snapshotPrefix: etcd-snapshot, directory: /var/lib/rancher/rke2/server/db/snapshots"
Nov 16 07:14:03 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:14:03Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700116202"
Nov 16 07:14:03 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:14:03Z" level=info msg="Saving etcd snapshot etcd-snapshot-testissue-pool1-697442e0-br5np-1700118842 to S3"
Nov 16 07:14:03 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:14:03Z" level=info msg="Uploading snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700118842 to S3"
Nov 16 07:14:04 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:14:04Z" level=info msg="S3 upload complete for etcd-snapshot-testissue-pool1-697442e0-br5np-1700118842"
Nov 16 07:14:04 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:14:04Z" level=info msg="Applying snapshot retention policy to snapshots stored in S3: retention: 10, snapshotPrefix: test/etcd-snapshot"
Nov 16 07:14:04 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:14:04Z" level=info msg="Removing S3 snapshot: test/etcd-snapshot-testissue-pool1-697442e0-br5np-1700115723"
Nov 16 07:14:04 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:14:04Z" level=error msg="failed to save snapshot data to configmap: ConfigMap "rke2-etcd-snapshots" is invalid: []: Too long: must have at most 1048576 bytes"
^C
root@testissue-pool1-697442e0-br5np:~# journalctl -u rke2-server -f
-- Logs begin at Thu 2023-11-16 04:32:30 UTC. --
Nov 16 07:14:03 testissue-pool1-697442e0-br5np rke2[92866]: {"level":"info","ts":"2023-11-16T07:14:03.215276Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"30 MB","took":"now"}
Nov 16 07:14:03 testissue-pool1-697442e0-br5np rke2[92866]: {"level":"info","ts":"2023-11-16T07:14:03.215404Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700118842"}
Nov 16 07:14:03 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:14:03Z" level=info msg="Applying local snapshot retention policy: retention: 10, snapshotPrefix: etcd-snapshot, directory: /var/lib/rancher/rke2/server/db/snapshots"
Nov 16 07:14:03 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:14:03Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700116202"
Nov 16 07:14:03 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:14:03Z" level=info msg="Saving etcd snapshot etcd-snapshot-testissue-pool1-697442e0-br5np-1700118842 to S3"
Nov 16 07:14:03 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:14:03Z" level=info msg="Uploading snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700118842 to S3"
Nov 16 07:14:04 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:14:04Z" level=info msg="S3 upload complete for etcd-snapshot-testissue-pool1-697442e0-br5np-1700118842"
Nov 16 07:14:04 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:14:04Z" level=info msg="Applying snapshot retention policy to snapshots stored in S3: retention: 10, snapshotPrefix: test/etcd-snapshot"
Nov 16 07:14:04 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:14:04Z" level=info msg="Removing S3 snapshot: test/etcd-snapshot-testissue-pool1-697442e0-br5np-1700115723"
Nov 16 07:14:04 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:14:04Z" level=error msg="failed to save snapshot data to configmap: ConfigMap "rke2-etcd-snapshots" is invalid: []: Too long: must have at most 1048576 bytes"
Nov 16 07:16:00 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:16:00Z" level=info msg="wake, now=2023-11-16T07:16:00Z"
Nov 16 07:16:00 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:16:00Z" level=info msg="run, now=2023-11-16T07:16:00Z, entry=1, next=2023-11-16T07:18:00Z"
Nov 16 07:16:02 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:16:02Z" level=info msg="Saving etcd snapshot to /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700118962"
Nov 16 07:16:02 testissue-pool1-697442e0-br5np rke2[92866]: {"level":"info","ts":"2023-11-16T07:16:02.772373Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700118962.part"}
Nov 16 07:16:02 testissue-pool1-697442e0-br5np rke2[92866]: {"level":"info","ts":"2023-11-16T07:16:02.784457Z","logger":"client","caller":"[email protected]/maintenance.go:212","msg":"opened snapshot stream; downloading"}
Nov 16 07:16:02 testissue-pool1-697442e0-br5np rke2[92866]: {"level":"info","ts":"2023-11-16T07:16:02.784653Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
Nov 16 07:16:03 testissue-pool1-697442e0-br5np rke2[92866]: {"level":"info","ts":"2023-11-16T07:16:03.544381Z","logger":"client","caller":"[email protected]/maintenance.go:220","msg":"completed snapshot read; closing"}
Nov 16 07:16:03 testissue-pool1-697442e0-br5np rke2[92866]: {"level":"info","ts":"2023-11-16T07:16:03.632944Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"33 MB","took":"now"}
Nov 16 07:16:03 testissue-pool1-697442e0-br5np rke2[92866]: {"level":"info","ts":"2023-11-16T07:16:03.63337Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-testissue-pool1-697442e0-br5np-1700118962"}
Nov 16 07:16:03 testissue-pool1-697442e0-br5np rke2[92866]: time="2023-11-16T07:16:03Z" level=error msg="failed to save local snapshot data to configmap: ConfigMap "rke2-etcd-snapshots" is invalid: []: Too long: must have at most 1048576 bytes"

`

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@brandond brandond

@aganesh-suse aganesh-suse

Labels
None yet
Projects
None yet
Milestone
v1.25.15+rke2r1
Development

No branches or pull requests
4 participants
@brandond @sadhugit @caroline-suse-rancher @aganesh-suse