Emit events for certificates about to expire

[brandond]

RKE2 tracking issue for

• Emit events for certificates about to expire k3s-io/k3s#9742

[ShylajaDevadiga]

Validated using latest commit id 95e13dc on master branch

Environment Details

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:

cat /etc/os-release
NAME="SLES"
VERSION="15-SP5"
VERSION_ID="15.5"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"

Cluster Configuration:
1 server 1 agent node

Config.yaml:

cat /etc/rancher/rke2/config.yaml
write-kubeconfig-mode: "0644"
token: <TOKEN>

Steps to reproduce the issue and validate the fix

1. Copy config.yaml
2. Set env variable CATTLE_NEW_SIGNED_CERT_EXPIRATION_DAYS=30 in /etc/default/rke2-server file
3. Install rke2
4. Check the warning when the certs are within 90 days of expiring

Validation results:

Details

rke2 -v
rke2 version v1.29.3+dev.95e13dc6 (95e13dc)
go version go1.21.8 X:boringcrypto

ec2-user@ip-172-31-1-234:~> kubectl get event|grep -i cert 3m59s Warning CertificateExpirationWarning node/ip-172-31-1-234 Node certificates require attention - restart rke2 on this node to trigger automatic rotation: rke2-controller/client-rke2-controller.crt: certificate CN=system:rke2-controller will expire within 90 days at 2024-05-15T23:26:07Z, rke2-controller/client-rke2-controller.crt: certificate CN=system:rke2-controller will expire within 90 days at 2024-05-15T23:26:07Z, auth-proxy/client-auth-proxy.crt: certificate CN=system:auth-proxy will expire within 90 days at 2024-05-15T23:26:07Z, cloud-controller/client-rke2-cloud-controller.crt: certificate CN=rke2-cloud-controller-manager will expire within 90 days at 2024-05-15T23:26:07Z, etcd/client.crt: certificate CN=etcd-client will expire within 90 days at 2024-05-15T23:26:07Z, etcd/server-client.crt: certificate CN=etcd-server will expire within 90 days at 2024-05-15T23:26:07Z, etcd/peer-server-client.crt: certificate CN=etcd-peer will expire within 90 days at 2024-05-15T23:26:07Z, scheduler/client-scheduler.crt: certificate CN=system:kube-scheduler will expire within 90 days at 2024-05-15T23:26:07Z, supervisor/client-supervisor.crt: certificate CN=system:rke2-supervisor,O=system:masters will expire within 90 days at 2024-05-15T23:26:07Z, kube-proxy/client-kube-proxy.crt: certificate CN=system:kube-proxy will expire within 90 days at 2024-05-15T23:26:07Z, kube-proxy/client-kube-proxy.crt: certificate CN=system:kube-proxy will expire within 90 days at 2024-05-15T23:26:07Z, kubelet/client-kubelet.crt: certificate CN=system:node:ip-172-31-1-234,O=system:nodes will expire within 90 days at 2024-05-15T23:26:09Z, kubelet/serving-kubelet.crt: certificate CN=ip-172-31-1-234 will expire within 90 days at 2024-05-15T23:26:09Z, api-server/client-kube-apiserver.crt: certificate CN=system:apiserver,O=system:masters will expire within 90 days at 2024-05-15T23:26:07Z, api-server/serving-kube-apiserver.crt: certificate CN=kube-apiserver will expire within 90 days at 2024-05-15T23:26:07Z, admin/client-admin.crt: certificate CN=system:admin,O=system:masters will expire within 90 days at 2024-05-15T23:26:07Z, controller-manager/client-controller.crt: certificate CN=system:kube-controller-manager will expire within 90 days at 2024-05-15T23:26:07Z

ec2-user@ip-172-31-1-234:~> sudo /usr/local/bin/rke2 certificate check INFO[0000] Server detected, checking agent and server certificates INFO[0000] Checking certificates for admin WARN[0000] /var/lib/rancher/rke2/server/tls/client-admin.crt: certificate CN=system:admin,O=system:masters will expire within 90 days at 2024-05-15T23:26:07Z INFO[0000] /var/lib/rancher/rke2/server/tls/client-admin.crt: certificate CN=rke2-client-ca@1713223567 is ok, expires at 2034-04-13T23:26:07Z INFO[0000] Checking certificates for kube-proxy WARN[0000] /var/lib/rancher/rke2/server/tls/client-kube-proxy.crt: certificate CN=system:kube-proxy will expire within 90 days at 2024-05-15T23:26:07Z INFO[0000] /var/lib/rancher/rke2/server/tls/client-kube-proxy.crt: certificate CN=rke2-client-ca@1713223567 is ok, expires at 2034-04-13T23:26:07Z WARN[0000] /var/lib/rancher/rke2/agent/client-kube-proxy.crt: certificate CN=system:kube-proxy will expire within 90 days at 2024-05-15T23:26:07Z INFO[0000] /var/lib/rancher/rke2/agent/client-kube-proxy.crt: certificate CN=rke2-client-ca@1713223567 is ok, expires at 2034-04-13T23:26:07Z INFO[0000] Checking certificates for supervisor WARN[0000] /var/lib/rancher/rke2/server/tls/client-supervisor.crt: certificate CN=system:rke2-supervisor,O=system:masters will expire within 90 days at 2024-05-15T23:26:07Z INFO[0000] /var/lib/rancher/rke2/server/tls/client-supervisor.crt: certificate CN=rke2-client-ca@1713223567 is ok, expires at 2034-04-13T23:26:07Z INFO[0000] Checking certificates for kubelet WARN[0000] /var/lib/rancher/rke2/agent/client-kubelet.crt: certificate CN=system:node:ip-172-31-1-234,O=system:nodes will expire within 90 days at 2024-05-15T23:26:09Z INFO[0000] /var/lib/rancher/rke2/agent/client-kubelet.crt: certificate CN=rke2-client-ca@1713223567 is ok, expires at 2034-04-13T23:26:07Z WARN[0000] /var/lib/rancher/rke2/agent/serving-kubelet.crt: certificate CN=ip-172-31-1-234 will expire within 90 days at 2024-05-15T23:26:09Z INFO[0000] /var/lib/rancher/rke2/agent/serving-kubelet.crt: certificate CN=rke2-server-ca@1713223567 is ok, expires at 2034-04-13T23:26:07Z INFO[0000] Checking certificates for api-server WARN[0000] /var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt: certificate CN=system:apiserver,O=system:masters will expire within 90 days at 2024-05-15T23:26:07Z INFO[0000] /var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt: certificate CN=rke2-client-ca@1713223567 is ok, expires at 2034-04-13T23:26:07Z WARN[0000] /var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt: certificate CN=kube-apiserver will expire within 90 days at 2024-05-15T23:26:07Z INFO[0000] /var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt: certificate CN=rke2-server-ca@1713223567 is ok, expires at 2034-04-13T23:26:07Z INFO[0000] Checking certificates for auth-proxy WARN[0000] /var/lib/rancher/rke2/server/tls/client-auth-proxy.crt: certificate CN=system:auth-proxy will expire within 90 days at 2024-05-15T23:26:07Z INFO[0000] /var/lib/rancher/rke2/server/tls/client-auth-proxy.crt: certificate CN=rke2-request-header-ca@1713223567 is ok, expires at 2034-04-13T23:26:07Z INFO[0000] Checking certificates for cloud-controller WARN[0000] /var/lib/rancher/rke2/server/tls/client-rke2-cloud-controller.crt: certificate CN=rke2-cloud-controller-manager will expire within 90 days at 2024-05-15T23:26:07Z INFO[0000] /var/lib/rancher/rke2/server/tls/client-rke2-cloud-controller.crt: certificate CN=rke2-client-ca@1713223567 is ok, expires at 2034-04-13T23:26:07Z INFO[0000] Checking certificates for controller-manager WARN[0000] /var/lib/rancher/rke2/server/tls/client-controller.crt: certificate CN=system:kube-controller-manager will expire within 90 days at 2024-05-15T23:26:07Z INFO[0000] /var/lib/rancher/rke2/server/tls/client-controller.crt: certificate CN=rke2-client-ca@1713223567 is ok, expires at 2034-04-13T23:26:07Z INFO[0000] Checking certificates for etcd WARN[0000] /var/lib/rancher/rke2/server/tls/etcd/client.crt: certificate CN=etcd-client will expire within 90 days at 2024-05-15T23:26:07Z INFO[0000] /var/lib/rancher/rke2/server/tls/etcd/client.crt: certificate CN=etcd-server-ca@1713223567 is ok, expires at 2034-04-13T23:26:07Z WARN[0000] /var/lib/rancher/rke2/server/tls/etcd/server-client.crt: certificate CN=etcd-server will expire within 90 days at 2024-05-15T23:26:07Z INFO[0000] /var/lib/rancher/rke2/server/tls/etcd/server-client.crt: certificate CN=etcd-server-ca@1713223567 is ok, expires at 2034-04-13T23:26:07Z WARN[0000] /var/lib/rancher/rke2/server/tls/etcd/peer-server-client.crt: certificate CN=etcd-peer will expire within 90 days at 2024-05-15T23:26:07Z INFO[0000] /var/lib/rancher/rke2/server/tls/etcd/peer-server-client.crt: certificate CN=etcd-peer-ca@1713223567 is ok, expires at 2034-04-13T23:26:07Z INFO[0000] Checking certificates for scheduler WARN[0000] /var/lib/rancher/rke2/server/tls/client-scheduler.crt: certificate CN=system:kube-scheduler will expire within 90 days at 2024-05-15T23:26:07Z INFO[0000] /var/lib/rancher/rke2/server/tls/client-scheduler.crt: certificate CN=rke2-client-ca@1713223567 is ok, expires at 2034-04-13T23:26:07Z INFO[0000] Checking certificates for rke2-controller WARN[0000] /var/lib/rancher/rke2/server/tls/client-rke2-controller.crt: certificate CN=system:rke2-controller will expire within 90 days at 2024-05-15T23:26:07Z INFO[0000] /var/lib/rancher/rke2/server/tls/client-rke2-controller.crt: certificate CN=rke2-client-ca@1713223567 is ok, expires at 2034-04-13T23:26:07Z WARN[0000] /var/lib/rancher/rke2/agent/client-rke2-controller.crt: certificate CN=system:rke2-controller will expire within 90 days at 2024-05-15T23:26:07Z INFO[0000] /var/lib/rancher/rke2/agent/client-rke2-controller.crt: certificate CN=rke2-client-ca@1713223567 is ok, expires at 2034-04-13T23:26:07Z

ec2-user@ip-172-31-1-234:~> kubectl get --raw /api/v1/nodes/ip-172-31-1-234/proxy/metrics|grep expiration # HELP apiserver_client_certificate_expiration_seconds [ALPHA] Distribution of the remaining lifetime on the certificate used to authenticate a request. # TYPE apiserver_client_certificate_expiration_seconds histogram apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0 apiserver_client_certificate_expiration_seconds_bucket{le="1800"} 0 apiserver_client_certificate_expiration_seconds_bucket{le="3600"} 0 apiserver_client_certificate_expiration_seconds_bucket{le="7200"} 0 apiserver_client_certificate_expiration_seconds_bucket{le="21600"} 0 apiserver_client_certificate_expiration_seconds_bucket{le="43200"} 0 apiserver_client_certificate_expiration_seconds_bucket{le="86400"} 0 apiserver_client_certificate_expiration_seconds_bucket{le="172800"} 0 apiserver_client_certificate_expiration_seconds_bucket{le="345600"} 0 apiserver_client_certificate_expiration_seconds_bucket{le="604800"} 0 apiserver_client_certificate_expiration_seconds_bucket{le="2.592e+06"} 1 apiserver_client_certificate_expiration_seconds_bucket{le="7.776e+06"} 1 apiserver_client_certificate_expiration_seconds_bucket{le="1.5552e+07"} 1 apiserver_client_certificate_expiration_seconds_bucket{le="3.1104e+07"} 1 apiserver_client_certificate_expiration_seconds_bucket{le="+Inf"} 1 apiserver_client_certificate_expiration_seconds_sum 2.591697539218925e+06 apiserver_client_certificate_expiration_seconds_count 1 ec2-user@ip-172-31-1-234:~>

[brandond]

Certificate lifetime metrics are not currently available in RKE2 due to lack of a metrics endpoint in the supervisor, tracking this in #5786