Secrets Encryption Test Plan

K3s currently has only a single command for managing secrets encryption - the --secrets-encryption flag. If this is turned on, a secrets encryption configuration document is generated using a random aescbckey key; this document is stored in the bootstrap data and shared among server nodes via the kube-apiserver --encryption-provider-config CLI flag. There are no provisions for rotating the key, disabling encryption after enabling it, etc. This feature expansion will allow for rotating the encryption key via a multi-step process in a single and multi node cluster. Ref: Design Doc

Feature

secrets-encryption keys rotation and enable/disable of encryption. The is all controlled via a new sub command k3s secrets-encrypt. Ref: Ticket

$ k3s secrets-encrypt --help
NAME:
   k3s secrets-encrypt - Control secrets encryption and keys rotation

USAGE:
   k3s secrets-encrypt command [command options] [arguments...]

COMMANDS:
   status     Print current status of secrets encryption
   enable     Enable secrets encryption
   disable    Disable secrets encryption
   prepare    Prepare for encryption keys rotation
   rotate     Rotate secrets encryption keys
   reencrypt  Reencrypt all data with new encryption key

OPTIONS:
   --help, -h  show help

Note: Stopping/killing and Restarting the service after each command, except status, is required. If not performed then the cluster goes in to bad state

Test Cases

Pre-requisite

Install k3s with `--secrets-encryption` flag

$ curl -sfL https://get.k3s.io | INSTALL_K3S_COMMIT=<commithash> INSTALL_K3S_EXEC="server --write-kubeconfig-mode 644 --secrets-encryption" sh -

OR

Install k3s using parameter `INSTALL_K3S_SKIP_START=true`. Start the service with `--secrets-encryption` flag

$ curl -sfL https://get.k3s.io | INSTALL_K3S_COMMIT=<commithash> INSTALL_K3S_EXEC="server --write-kubeconfig-mode 644" INSTALL_K3S_SKIP_START=true sh -
$ sudo k3s server secrets-encrypt

Adding secret

$ kubectl create secret generic secret1 -n default --from-literal=mykey=mydata
secret/secret1 created
$ kubectl describe secret secret1 -n default
Name:         secret1
Namespace:    default
Labels:       <none>
Annotations:  <none>
Type:  Opaque
Data
====
mykey:  6 bytes

#	Test Case	Steps	Docs Needed?	Expected Result	Pass/Fail Version	Notes /GH issues
1	secrets-encrypt status Print current status of secrets encryption	# Check status $ sudo k3s secrets-encrypt status	Yes	Should show the status of secrets-encryption Encryption Status: Enabled Current Rotation Stage: start Server Encryption Hashes: All hashes match Active Key Type Name ------ -------- ---- * AES-CBC aescbckey	PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1	N/A
2	secrets-encrypt enable (Enabled by default) Enable secrets encryption Use flag --force --skip to properly enable	# Check status $ sudo k3s secrets-encrypt status # Enable secrets encryption $ sudo k3s secrets-encrypt enable # Check status $ sudo k3s secrets-encrypt status # Stop/Kill and Restart k3s service $ sudo systemctl stop k3s && sudo systemctl restart k3s	Yes	Console output shows secrets-encryption enabled Status should show the correct state of secrets-encryption Encryption Status: Enabled Current Rotation Stage: start Server Encryption Hashes: All hashes match Active Key Type Name ------ -------- ---- * AES-CBC aescbckey	PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1	N/A
3	secrets-encrypt disable Disable secrets encryption Use flag --force --skip to properly disable	# Check status, ensure it is enabled $ sudo k3s secrets-encrypt status # Disable secrets encryption $ sudo k3s secrets-encrypt disable # Check status $ sudo k3s secrets-encrypt status # Stop/Kill and Restart k3s service $ sudo systemctl stop k3s && sudo systemctl restart k3s # Deploy another secret and ensure it is not encrypted $ kubectl create secret generic secret2 -n default --from-literal=mykey=mydata	Yes	Console output shows secrets-encryption disabled Status should show the correct state of secrets-encryption Encryption Status: Disabled Current Rotation Stage: start Server Encryption Hashes: All hashes match Active Key Type Name ------ -------- ---- * AES-CBC aescbckey Use ETCDCTL to show secrets being encrypted/not encrypted. Change the below command to point to secret1, which should still show k8s:enc:aescbc:v1: and not have the data of the secret, meaning it is encrypted. secret2 will not show that, and will show the secret data, meaning it is not encrypted sudo ETCDCTL_API=3 etcdctl --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --endpoints https://127.0.0.1:2379 --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt get /registry/secrets/default/secret2 | hexdump -C	PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1
4	secrets-encrypt prepare Prepare for encryption keys rotation	# Check status, ensure its enabled $ sudo k3s secrets-encrypt status # Prepare secrets encryption $ sudo k3s secrets-encrypt prepare # Check status $ sudo k3s secrets-encrypt status # Stop/Kill and Restart k3s service $ sudo systemctl stop k3s && sudo systemctl restart k3s	Yes	Console output shows prepare completed successfully Status should show the correct state of secrets-encryption along with new key Encryption Status: Enabled Current Rotation Stage: prepare Server Encryption Hashes: All hashes match Active Key Type Name ------ -------- ---- * AES-CBC aescbckey AES-CBC aescbckey-2021-12-08T21:34:03Z	PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1
5	secrets-encrypt rotate Rotate secrets encryption keys	# Check status, ensure its enabled $ sudo k3s secrets-encrypt status # Prepare secrets encryption $ sudo k3s secrets-encrypt prepare # Check status, ensure it is in prepare stage $ sudo k3s secrets-encrypt status # Stop/Kill and Restart k3s service $ sudo systemctl stop k3s && sudo systemctl restart k3s # Rotate secrets encryption $ sudo k3s secrets-encrypt rotate # Check status $ sudo k3s secrets-encrypt status # Stop/Kill and Restart k3s service $ sudo systemctl stop k3s && sudo systemctl restart k3s	Yes	Console output shows rotate completed successfully Status should show the correct state of secrets-encryption with new key as active Encryption Status: Enabled Current Rotation Stage: rotate Server Encryption Hashes: All hashes match Active Key Type Name ------ -------- ---- * AES-CBC aescbckey-2021-12-08T21:34:03Z AES-CBC aescbckey	PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1
6	secrets-encrypt reencrypt Reencrypt all data with new encryption key Use flag -f or --force	# Check status, ensure its enabled $ sudo k3s secrets-encrypt status # Prepare secrets encryption $ sudo k3s secrets-encrypt prepare # Check status, ensure it is in prepare stage $ sudo k3s secrets-encrypt status # Stop/Kill and Restart k3s service $ sudo systemctl stop k3s && sudo systemctl restart k3s # Rotate secrets encryption $ sudo k3s secrets-encrypt rotate # Check status, ensure it is in rotate stage $ sudo k3s secrets-encrypt status # Stop/Kill and Restart k3s service $ sudo systemctl stop k3s && sudo systemctl restart k3s # Re-encrypt secrets encryption $ sudo k3s secrets-encrypt reencrypt # Check status, ensure it is in reencrypt_active stage $ sudo k3s secrets-encrypt status # Check status until it is in reencrypt_finished stage $ sudo k3s secrets-encrypt status # Stop/Kill and Restart k3s service $ sudo systemctl stop k3s && sudo systemctl restart k3s	Yes	Console output shows reencryption started Status should show the correct state of secrets-encryption with new key as active Encryption Status: Enabled Current Rotation Stage: reencrypt_active Server Encryption Hashes: All hashes match Active Key Type Name ------ -------- ---- * AES-CBC aescbckey-2021-12-08T21:34:03Z AES-CBC aescbckey Status shows correct state of secrets-encryption with new key as active and old key as removed Encryption Status: Enabled Current Rotation Stage: reencrypt_finished Server Encryption Hashes: All hashes match Active Key Type Name ------ -------- ---- * AES-CBC aescbckey-2021-12-08T21:34:03Z	PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1

Multi-node cluster (HA)

With embedded etcd

#	Test Case	Steps	Expected Result	Pass/Fail Version	Notes/GH issue
1	Perform secrets encryption rotation on multi node cluster with 3 servers, 1 agent	# Start up 3 K3s server $ k3s server --secrets-encryption # Check status on all the servers, ensure all shows the same result $ sudo k3s secrets-encrypt status # Select ONE server (S1 going forward) to perform the rotate on (doing any stage on any server is supported) # Perform secrets-encrypt prepare on S1 $ sudo k3s secrets-encrypt prepare # Stop/Kill and Restart k3s service on S1 $ sudo systemctl stop k3s && sudo systemctl restart k3s # Once S1 is back up, restart the other two servers $ sudo systemctl stop k3s && sudo systemctl restart k3s # Check status on all the servers, ensure all shows the same result $ sudo k3s secrets-encrypt status # Perform secrets-encrypt rotate on S1 $ sudo k3s secrets-encrypt rotate # Stop/Kill and Restart k3s service on S1 $ sudo systemctl stop k3s && sudo systemctl restart k3s # Once S1 is back up, restart the other two servers $ sudo systemctl stop k3s && sudo systemctl restart k3s # Check status on all the servers, ensure all shows the same result $ sudo k3s secrets-encrypt status # Perform secrets-encrypt reencrypt on S1 $ sudo k3s secrets-encrypt reencrypt # Check status on all the servers $ sudo k3s secrets-encrypt status # Stop/Kill and Restart k3s service on S1 $ sudo systemctl stop k3s && sudo systemctl restart k3s # Once S1 is back up, restart the other two servers $ sudo systemctl stop k3s && sudo systemctl restart k3s # Check status on all the servers, ensure all shows the same result $ sudo k3s secrets-encrypt status		PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1
2	Ensure all server nodes perform reconciliation on restart.	Start 3 k3s servers (1 cluster-init, 2 --server) On server 2 run k3s secrets-encrypt prepare Check that the status has updated only on server 2 k3s secrets-encrypt status Kill all 3 servers Restart only servers 2 and 3. Check that server 3 now has the same status as server 2 k3s secrets-encrypt status		PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1	Detailed steps to be added later

With external DB

#	Test Case	Steps	Expected Result	Pass/Fail Version	Notes/GH issue
1	Perform secrets encryption rotation on multi node cluster with 3 servers, 1 agent	# Start up 3 K3s server joined to external DB (Ex: mySQL) $ k3s server --secrets-encryption --datastore-endpoint "mysql://root:mysql@tcp(<server_ip>:3306)/k3s" # Check status on all the servers, ensure all shows the same result $ sudo k3s secrets-encrypt status # Select ONE server (S1 going forward) to perform the rotate on (doing any stage on any server is supported) # Perform secrets-encrypt prepare on S1 $ sudo k3s secrets-encrypt prepare # Stop/Kill and Restart k3s service on S1 $ sudo systemctl stop k3s && sudo systemctl restart k3s # Once S1 is back up, restart the other two servers $ sudo systemctl stop k3s && sudo systemctl restart k3s # Check status on all the servers, ensure all shows the same result $ sudo k3s secrets-encrypt status # Perform secrets-encrypt rotate on S1 $ sudo k3s secrets-encrypt rotate # Stop/Kill and Restart k3s service on S1 $ sudo systemctl stop k3s && sudo systemctl restart k3s # Once S1 is back up, restart the other two servers $ sudo systemctl stop k3s && sudo systemctl restart k3s # Check status on all the servers, ensure all shows the same result $ sudo k3s secrets-encrypt status # Perform secrets-encrypt reencrypt on S1 $ sudo k3s secrets-encrypt reencrypt # Check status on all the servers $ sudo k3s secrets-encrypt status # Stop/Kill and Restart k3s service on S1 $ sudo systemctl stop k3s && sudo systemctl restart k3s # Once S1 is back up, restart the other two servers $ sudo systemctl stop k3s && sudo systemctl restart k3s # Check status on all the servers, ensure all shows the same result $ sudo k3s secrets-encrypt status		PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1

Additional Scenarios

#	Test Scenarios	Steps	Expected Result	Pass/Fail Version	Notes/GH issues
1	Perform commands without restarting k3s	# Prepare secrets encryption $ sudo k3s secrets-encrypt prepare # Rotate secrets encryption $ sudo k3s secrets-encrypt rotate # Re-encrypt secrets encryption $ sudo k3s secrets-encrypt reencrypt # Stop/Kill and Restart k3s service $ sudo systemctl stop k3s && sudo systemctl restart k3s	The system does not restart and goes in to infinite loop with the error below Dec 08 21:39:09 ip-172-31-41-171 k3s[5579]: E1208 21:39:09.263197 5579 cacher.go:420] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/default-token-f6bkz": no matching key was found for the provided AES transformer; reinitializing... Dec 08 21:39:09 ip-172-31-41-171 k3s[5579]: E1208 21:39:09.941337 5579 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/default-token-f6bkz": no matching key was found for the provided AES transformer	PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1	This action breaks the cluster so don't try this! This happens because all the secrets gets locked and the key gets thrown away
2	Perform same command repeatedly / multiple times	On running `sudo k3s secrets-encrypt enable`	Console shows `secrets-encryption enabled` without any errors	PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1
		On running `sudo k3s secrets-encrypt disable`	Console shows `secrets-encryption enabled` without any errors	PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1
		On running `sudo k3s secrets-encrypt prepare`	Console shows `FATA[0000] https://127.0.0.1:6443/v1-k3s/encrypt/config: 400 Bad Request error secrets-encrypt-90363: see server logs for more info` after showing `prepare completed successfully` on the first command run	PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1
		On running `sudo k3s secrets-encrypt rotate`	Console shows `FATA[0000] https://127.0.0.1:6443/v1-k3s/encrypt/config: 400 Bad Request error secrets-encrypt-45155: see server logs for more info` after showing `rotate completed successfully` on the first command run	PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1
		On running `sudo k3s secrets-encrypt reencrypt`	Console shows `reencryption started` with no errors but the system gets stuck with rotation stage as `reencrypt_request` and spits out errors in the log Dec 10 18:28:15 ip-172-31-32-180 k3s[2146]: time="2021-12-10T18:28:15Z" level=error msg="error syncing 'ip-172-31-32-180': handler reencrypt-controller: invalid hash: 8d0f2a2ad73ce974c4cc39874ebcb3b31d142774aed2c4771e4b4205c94a028f found on node ip-172-31-32-180, requeuing"	PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1
3	Perform each stage on different servers in HA cluster			PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1
4	Perform commands out of sync	# Rotate secrets encryption $ sudo k3s secrets-encrypt rotate # Prepare secrets encryption $ sudo k3s secrets-encrypt prepare # Disable secrets encryption $ sudo k3s secrets-encrypt disable # Enable secrets encryption $ sudo k3s secrets-encrypt enable	Console shows FATA[0000] https://127.0.0.1:6443/v1-k3s/encrypt/config: 400 Bad Request error secrets-encrypt-5012: see server logs for more info	PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1
5	Perform re-encrypt with 1000+ secrets stored in the DB
6	Perform commands on agent node instead of server	Install k3s with 1 server and 1 agent node On the agent node, ran sudo k3s secrets-encrypt status	Console shows FATA[0000] open /var/lib/rancher/k3s/server/token: no such file or directory	PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1

    </div>