k3s certificate expired on worker node side

[rthamrin]

I read some discussions here but most of them failed during checking the pods

#kubectl get pods -A

I have some pods in my cluster, some are working fine but some are not, when I try to look up what was going on inside the pods, I figure out this message appears. x509: certificate has expired or is not yet valid: current time

this is only happening to the service/pod who connected to the worker-node, and I check the certificate validation, and I think it is expired already

# for i in `ls /var/lib/rancher/k3s/server/tls/*.crt`; do echo $i; openssl x509 -enddate -noout -in $i; done
/var/lib/rancher/k3s/server/tls/client-admin.crt
notAfter=Oct 25 14:16:41 2022 GMT
/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt
notAfter=Oct 25 14:16:41 2022 GMT
/var/lib/rancher/k3s/server/tls/client-ca.crt
notAfter=Oct 23 14:16:41 2031 GMT
/var/lib/rancher/k3s/server/tls/client-controller.crt
notAfter=Oct 25 14:16:41 2022 GMT
/var/lib/rancher/k3s/server/tls/client-k3s-cloud-controller.crt
notAfter=Oct 25 14:16:41 2022 GMT
/var/lib/rancher/k3s/server/tls/client-k3s-controller.crt
notAfter=Oct 25 14:16:41 2022 GMT
/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt
notAfter=Oct 25 14:16:41 2022 GMT
/var/lib/rancher/k3s/server/tls/client-kube-proxy.crt
notAfter=Oct 25 14:16:41 2022 GMT
/var/lib/rancher/k3s/server/tls/client-scheduler.crt
notAfter=Oct 25 14:16:41 2022 GMT
/var/lib/rancher/k3s/server/tls/request-header-ca.crt
notAfter=Oct 23 14:16:41 2031 GMT
/var/lib/rancher/k3s/server/tls/server-ca.crt
notAfter=Oct 23 14:16:41 2031 GMT
/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt
notAfter=Dec  3 06:43:15 2022 GMT
root@Master:/var/lib/rancher/k3s/server#

Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  47m                  default-scheduler  Successfully assigned ingress/nginx-deployment-9fd77fb68-tqptj to k3s-worker-04
  Warning  Failed     24d                  kubelet            Failed to pull image "nginx:1.19.0": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/nginx:1.19.0": failed to resolve reference "docker.io/library/nginx:1.19.0": failed to do request: Head "https://registry-1.docker.io/v2/library/nginx/manifests/1.19.0": x509: certificate has expired or is not yet valid: current time 2021-11-10T12:28:50+08:00 is before 2021-11-30T00:00:00Z

my questions are

1. how to renew it? I tried to update the k3s version to k3s version v1.21.7+k3s1 for master node and worker nodes but seems not working. the certificate still remain the same
2. is it a worker node problem? Is the certificate problem on the worker's side?
3. I tried to follow these steps but its nothing happen

ps: I don't use docker in my cluster or rancher as well

[rthamrin]

I got the solution thank you.

[ChristianCiach]

Would you please post your solution for other people who may find this discussion via Google?

[rthamrin]

I tried to understand these links.

https://www.ibm.com/support/pages/node/6444205
rancher/rancher#26984
https://tienbm90.medium.com/how-to-renew-rancher-certificates-when-expired-cf8f05942eac

check the master expired date cert
for i in `ls /var/lib/rancher/k3s/server/tls/*.crt`; do echo $i; openssl x509 -enddate -noout -in $i; done

Today is "12 Dec 2021"
my steps are :

we have to stop the time
timedatectl set-ntp off

set the time to forward date, I was set mine on 25 may 2022
date --set="2022-05-25 03:00:00.000"

remove the dynamic-cert, you can check all your cert here /var/lib/rancher/k3s/server/tls
rm /var/lib/rancher/k3s/server/tls/dynamic-cert.json

remove the k3s-serving cert
kubectl delete secret -n kube-system k3s-serving

set the time back on
timedatectl set-ntp on

restart your K3S
systemctl restart k3s

once you restart the K3S your certificate will re-create the new certificate

I am not sure if I update my master cert means automatically cert on my K3S-agent will renew, so I removed all my K3S-agent cert. Go to your agent-worker I am using linux arch. check your certificate on the agent side

for i in `ls /var/lib/rancher/k3s/agent/*.crt`; do echo $i; openssl x509 -enddate -noout -in $i; done

I remove all my certificate on agent side
rm /var/lib/rancher/k3s/agent/*.crt
restart my k3s-agent
kubectl restart k3s-agent

[blecx]

Thank you very much for sharing this. Great