[Release-1.28] - secrets-encrypt rotate-keys is not working since the metics server output is not as expected

[dereknola]

Backport fix for secrets-encrypt rotate-keys is not working since the metics server output is not as expected

• secrets-encrypt rotate-keys is not working since the metics server output is not as expected #5535

[ShylajaDevadiga]

Validated on release-1.28 branch with commit df95237

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

> cat /etc/os-release
NAME="SLES"
VERSION="15-SP5"
VERSION_ID="15.5"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"

Cluster Configuration:

HA : 1 etcd , 2 cp, 1 agent node

Config.yaml:

ETCD server config:

token: xxxx
disable-apiserver: true
disable-controller-manager: true
disable-scheduler: true
write-kubeconfig-mode: "0644"
secrets-encryption: true
node-external-ip: 1.1.1.1
debug: true

CP only node configs:

token: xxxx
server: https://1.1.1.1:9345
disable-etcd: true
write-kubeconfig-mode: "0644"
secrets-encryption: true
node-external-ip: 1.2.3.4
debug: true

Steps to reproduce:

1. Copy config.yaml

$ sudo mkdir -p /etc/rancher/rke2 && sudo cp config.yaml /etc/rancher/rke2

2. Install RKE2

curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_COMMIT='df952375a60c353d2eb8690fc96fa1aa6e681f65' INSTALL_RKE2_CHANNEL=testing  sh -

3. Start the RKE2 service

$ sudo systemctl enable --now rke2-server
or 
$ sudo systemctl enable --now rke2-agent

4. Verify Cluster Status:

kubectl get nodes -o wide
kubectl get pods -A

5. Run secrets-encrypt rotate-keys:

sudo rke2 secrets-encrypt rotate-keys

6. Restart rke2 services on etcd node (leader node) first and then rest of the cp nodes.

sudo rke2 secrets-encrypt status

Validation Results:

• rke2 version used for validation:

> rke2 -v
rke2 version v1.28.9-rc3+rke2r1 (df952375a60c353d2eb8690fc96fa1aa6e681f65)

> kubectl get nodes
NAME              STATUS   ROLES                  AGE     VERSION
ip-172-31-0-97    Ready    control-plane,master   10m     v1.28.9+rke2r1
ip-172-31-10-96   Ready    control-plane,master   10m     v1.28.9+rke2r1
ip-172-31-15-78   Ready    etcd                   10m     v1.28.9+rke2r1
ip-172-31-8-86    Ready    <none>                 9m45s   v1.28.9+rke2r1

Rotate-keys:

ec2-user@ip-172-31-10-96:~> sudo /usr/local/bin/rke2 secrets-encrypt rotate-keys
keys rotated, reencryption started
ec2-user@ip-172-31-10-96:~> sudo /usr/local/bin/rke2 secrets-encrypt status
Encryption Status: Enabled
Current Rotation Stage: reencrypt_finished
Server Encryption Hashes: hash does not match between ip-172-31-0-97 and ip-172-31-10-96

Restart rke2 services on etcd node (leader node) first and then rest of the cp nodes.

> sudo /usr/local/bin/rke2 secrets-encrypt status
Encryption Status: Enabled
Current Rotation Stage: reencrypt_finished
Server Encryption Hashes: All hashes match

Active  Key Type  Name
------  --------  ----
 *      AES-CBC   aescbckey-2024-04-25T21:27:05Z