[Release-1.26] - Bump k3s for etcd s3 fixes #5072

brandond · 2023-11-21T23:09:37Z

Backport fix for Bump k3s for etcd s3 fixes

Bump k3s for etcd s3 fixes #5068

mdrahman-suse · 2023-11-28T22:46:07Z

Validated on version v1.26.11-rc1+rke2r1

https://github.com/k3s-io/k3s/issues/8918

Environment Details

Infrastructure

Cloud
Hosted

Node(s) CPU architecture, OS, and Version:

Ubuntu 22.04

Cluster Configuration:

1 server

Config.yaml:

write-kubeconfig-mode: 644
token: summerheat
node-name: server1
node-external-ip: <publicIP>
debug: true

Testing Steps

Copy config.yaml

$ sudo mkdir -p /etc/rancher/rke2 && sudo cp config.yaml /etc/rancher/rke2

Install RKE2
Perform rke2 etcd snapshot save on s3 with s3 prop as invalid data

sudo rke2 etcd-snapshot save   --s3    --s3-endpoint="invalid"  --s3-bucket="invalid"   --s3-folder="invalid"   --s3-access-key="invalid"    --s3-secret-key="invalid"    --s3-region="invalid"

Ensure the error is handled accordingly

Replication Results:

rke2 version used for replication:

rke2 version v1.26.10+rke2r2 (21e3a8c82da71473f2b846065dcab197a9b2c9d8)
go version go1.20.10 X:boringcrypto

Observed similar panic in almost all the invalid cases with a variation in WARN for specific invalid props

WARN[0000] Unable to initialize S3 client: Access Denied.
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x28cfa59]

goroutine 1 [running]:
github.com/k3s-io/k3s/pkg/etcd.(*S3).snapshotRetention(0xc0009d938a?, {0x3b7cb28?, 0xc0009d6dc0?})
	/go/pkg/mod/github.com/k3s-io/[email protected]/pkg/etcd/s3.go:284 +0x59
github.com/k3s-io/k3s/pkg/etcd.(*ETCD).Snapshot(0xc0003eea00, {0x3b7cb28, 0xc0009d6dc0})
	/go/pkg/mod/github.com/k3s-io/[email protected]/pkg/etcd/snapshot.go:375 +0x13ca
github.com/k3s-io/k3s/pkg/cli/etcdsnapshot.save(0xc0009d49a0, 0xc00097dbc8?)
	/go/pkg/mod/github.com/k3s-io/[email protected]/pkg/cli/etcdsnapshot/etcd_snapshot.go:127 +0x92
github.com/k3s-io/k3s/pkg/cli/etcdsnapshot.Save(0xc0009d49a0?)
	/go/pkg/mod/github.com/k3s-io/[email protected]/pkg/cli/etcdsnapshot/etcd_snapshot.go:110 +0x45
github.com/urfave/cli.HandleAction({0x2fde0c0?, 0x37d9da0?}, 0x4?)
	/go/pkg/mod/github.com/urfave/[email protected]/app.go:524 +0x50
github.com/urfave/cli.Command.Run({{0x3643998, 0x4}, {0x0, 0x0}, {0x0, 0x0, 0x0}, {0x36a5cc1, 0x22}, {0x0, ...}, ...}, ...)
	/go/pkg/mod/github.com/urfave/[email protected]/command.go:175 +0x67b
github.com/urfave/cli.(*App).RunAsSubcommand(0xc000753180, 0xc0009d46e0)
	/go/pkg/mod/github.com/urfave/[email protected]/app.go:405 +0xe87
github.com/urfave/cli.Command.startApp({{0x365b530, 0xd}, {0x0, 0x0}, {0x0, 0x0, 0x0}, {0x36a5cc1, 0x22}, {0x0, ...}, ...}, ...)
	/go/pkg/mod/github.com/urfave/[email protected]/command.go:380 +0xb7f
github.com/urfave/cli.Command.Run({{0x365b530, 0xd}, {0x0, 0x0}, {0x0, 0x0, 0x0}, {0x36a5cc1, 0x22}, {0x0, ...}, ...}, ...)
	/go/pkg/mod/github.com/urfave/[email protected]/command.go:103 +0x845
github.com/urfave/cli.(*App).Run(0xc000752fc0, {0xc0009f0780, 0xc, 0x14})
	/go/pkg/mod/github.com/urfave/[email protected]/app.go:277 +0xb87
main.main()
	/source/main.go:23 +0x97e

Validation Results:

rke2 version used for validation:

rke2 version v1.26.11-rc1+rke2r1 (3f9afa7c475e163ad96981d94ee25f120cc54298)
go version go1.20.11 X:boringcrypto

No panic observed for invalid s3 prop (accesskey/bucket-name)

INFO[0000] Checking if S3 bucket <bucket> exists
WARN[0000] Unable to initialize S3 client: Access Denied.
INFO[0000] Reconciling ETCDSnapshotFile resources
INFO[0000] Checking if S3 bucket <bucket> exists
WARN[0000] Unable to initialize S3 client: Access Denied.
INFO[0000] Reconciliation of ETCDSnapshotFile resources complete
FATA[0000] Access Denied.

$ kubectl get etcdsnapshotfile | grep s3-on-demand-server
s3-on-demand-server1-1701211461-41242b      on-demand-server1-1701211461   s3                                                                                        0         2023-11-28T22:44:21Z

$ kubectl get etcdsnapshotfile s3-on-demand-server1-1701211461-41242b -o yaml
apiVersion: k3s.cattle.io/v1
kind: ETCDSnapshotFile
metadata:
  creationTimestamp: "2023-11-28T22:44:21Z"
  finalizers:
  - wrangler.cattle.io/managed-etcd-snapshots-controller
  generation: 1
  labels:
    etcd.rke2.cattle.io/snapshot-storage-node: s3
  name: s3-on-demand-server1-1701211461-41242b
  resourceVersion: "1956"
  uid: 6b83c834-77c4-4368-95a3-3543cdb5732e
spec:
  location: ""
  nodeName: s3
  s3:
    bucket: <bucket>
    endpoint: s3.amazonaws.com
    prefix: rke2
    region: us-east-2
  snapshotName: on-demand-server1-1701211461
status:
  creationTime: "2023-11-28T22:44:21Z"
  error:
    message: Access Denied.
    time: "2023-11-28T22:44:21Z"
  readyToUse: false
  size: "0"

VestigeJ · 2023-11-29T22:52:58Z

##Environment Details
Validated using VERSION=v1.26.11+rke2r1

Infrastructure

Cloud

Node(s) CPU architecture, OS, and version:

Linux 5.14.21-150500.53-default x86_64 GNU/Linux
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"

Cluster Configuration:

NAME         STATUS   ROLES                       AGE   VERSION
ip-1-2-3-5   Ready    control-plane,etcd,master   17m   v1.26.11+rke2r1
ip-1-2-3-7   Ready    <none>                      14m   v1.26.11+rke2r1
ip-1-2-3-3   Ready    control-plane,etcd,master   15m   v1.26.11+rke2r1
ip-1-2-3-6   Ready    control-plane,etcd,master   16m   v1.26.11+rke2r1

Config.yaml:

server: https://1.2.3.5:9345
write-kubeconfig-mode: 644
debug: true
token: YOUR_TOKEN_HERE
profile: cis
selinux: true
node-external-ip: 1.2.3.3
etcd-s3: true
etcd-s3-bucket: "k3s-etcd-testing"
etcd-s3-endpoint: "yep.r2.cloudflarestorage.com"
etcd-s3-access-key: "hot"
etcd-s3-secret-key: "cold"

Validation Steps R2

$ curl https://get.rke2.io --output install-"rke2".sh
$ sudo chmod +x install-"rke2".sh
$ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd
$ sudo modprobe ip_vs_rr
$ sudo modprobe ip_vs_wrr
$ sudo modprobe ip_vs_sh
$ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/60-rke2-cis.conf
$ sudo cp 60-rke2-cis.conf /etc/sysctl.d/
$ sudo systemctl restart systemd-sysctl
$ sudo INSTALL_RKE2_VERSION=v1.26.11-rc1+rke2r1 INSTALL_RKE2_EXEC=server ./install-rke2.sh
$ sudo /usr/local/bin/rke2 etcd-snapshot save

Results:
$ sudo /usr/local/bin/rke2 etcd-snapshot save

WARN[0000] Unknown flag --server found in config.yaml, skipping
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --profile found in config.yaml, skipping
WARN[0000] Unknown flag --selinux found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
DEBU[0000] Attempting to retrieve extra metadata from rke2-etcd-snapshot-extra-metadata ConfigMap
DEBU[0000] Error encountered attempting to retrieve extra metadata from rke2-etcd-snapshot-extra-metadata ConfigMap, error: configmaps "rke2-etcd-snapshot-extra-metadata" not found
INFO[0000] Saving etcd snapshot to /var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-1-2-3-4-1701298001
{"level":"info","ts":"2023-11-29T22:46:40.853675Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-1-2-3-4-1701298001.part"}
{"level":"info","ts":"2023-11-29T22:46:40.857213Z","logger":"client","caller":"[email protected]/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-11-29T22:46:40.857253Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-11-29T22:46:40.986442Z","logger":"client","caller":"[email protected]/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-11-29T22:46:41.026764Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"12 MB","took":"now"}
{"level":"info","ts":"2023-11-29T22:46:41.026981Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-1-2-3-4-1701298001"}
INFO[0000] Checking if S3 bucket k3s-etcd-testing exists
INFO[0000] S3 bucket k3s-etcd-testing exists
INFO[0000] Saving etcd snapshot on-demand-ip-1-2-3-4-1701298001 to S3
INFO[0000] Uploading snapshot to s3://k3s-etcd-testing//var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-1-2-3-4-1701298001
INFO[0001] Uploaded snapshot metadata s3://k3s-etcd-testing/.metadata/on-demand-ip-1-2-3-4-1701298001
INFO[0001] S3 upload complete for on-demand-ip-1-2-3-4-1701298001
INFO[0001] Reconciling ETCDSnapshotFile resources
DEBU[0001] Found snapshotFile for on-demand-ip-1-2-3-4-1701294147 with key local-on-demand-ip-1-2-3-4-1701294147
DEBU[0001] Found snapshotFile for on-demand-ip-1-2-3-4-1701298001 with key local-on-demand-ip-1-2-3-4-1701298001
DEBU[0001] Found snapshotFile for on-demand-ip-1-2-3-4-1701294147 with key s3-on-demand-ip-1-2-3-4-1701294147
DEBU[0001] Found snapshotFile for on-demand-ip-1-2-3-4-1701298001 with key s3-on-demand-ip-1-2-3-4-1701298001
DEBU[0001] Found snapshotFile for on-demand-ip-1-2-3-4-1701294135 with key local-on-demand-ip-1-2-3-4-1701294135
DEBU[0001] Found ETCDSnapshotFile for on-demand-ip-1-2-3-4-1701294135 with key local-on-demand-ip-1-2-3-4-1701294135
DEBU[0001] Found ETCDSnapshotFile for on-demand-ip-1-2-3-4-1701294147 with key local-on-demand-ip-1-2-3-4-1701294147
DEBU[0001] Found ETCDSnapshotFile for on-demand-ip-1-2-3-4-1701298001 with key local-on-demand-ip-1-2-3-4-1701298001
DEBU[0001] Found ETCDSnapshotFile for on-demand-ip-1-2-3-4-1701294135 with key s3-on-demand-ip-1-2-3-4-1701294135
DEBU[0001] Found ETCDSnapshotFile for on-demand-ip-1-2-3-4-1701294147 with key s3-on-demand-ip-1-2-3-4-1701294147
DEBU[0001] Found ETCDSnapshotFile for on-demand-ip-1-2-3-4-1701298001 with key s3-on-demand-ip-1-2-3-4-1701298001
INFO[0001] Reconciliation of ETCDSnapshotFile resources complete

mdrahman-suse · 2023-11-30T19:15:50Z

Closing this but there is a nit fix that will be added in the next release which needs to be validated in rke2 and tracked in k3s: k3s-io/k3s#8925

brandond self-assigned this Nov 21, 2023

brandond added this to the v1.26.11+rke2r1 milestone Nov 21, 2023

brandond mentioned this issue Nov 21, 2023

[release-1.26] Bump K3s version for v1.26 #5074

Merged

mdrahman-suse assigned endawkins, VestigeJ and mdrahman-suse and unassigned endawkins Nov 22, 2023

mdrahman-suse closed this as completed Nov 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Release-1.26] - Bump k3s for etcd s3 fixes #5072

[Release-1.26] - Bump k3s for etcd s3 fixes #5072

brandond commented Nov 21, 2023

mdrahman-suse commented Nov 28, 2023

Environment Details

Testing Steps

VestigeJ commented Nov 29, 2023

Validation Steps R2

mdrahman-suse commented Nov 30, 2023

[Release-1.26] - Bump k3s for etcd s3 fixes #5072

[Release-1.26] - Bump k3s for etcd s3 fixes #5072

Comments

brandond commented Nov 21, 2023

mdrahman-suse commented Nov 28, 2023

Validated on version v1.26.11-rc1+rke2r1

Environment Details

Testing Steps

VestigeJ commented Nov 29, 2023

Validation Steps R2

mdrahman-suse commented Nov 30, 2023