Clarification about Custom CA certificates

[sushiMix]

According to the documentation : https://docs.k3s.io/cli/certificate#using-custom-ca-certificates, the certificates authorities should be used for the control plane.
When deploying with such configuration we find that:

• the k3s public port is using the provided CAs which is fine :)
• the local ports for controller-manager, cloud-controller-manager, kube-scheduler are NOT using such CA but self signed ones. Is it on purpose , the doc speaks about control plane?

[brandond]

Can you point to which specific certificates you're seeing as self-signed? In the default configuration, all of the certificates should be signed by one of the cluster CAs. Some of the Kubernetes components will generate their own self-signed certificates if you pass --cert-dir as an extra arg.

[sushiMix]

Hello, I have seen

• cloud-controller-manager
  Server certificate
  subject=CN = localhost@1692885233
  issuer=CN = localhost-ca@1692885233
• kube-scheduler
  Server certificate
  subject=CN = localhost@1692885234
  issuer=CN = localhost-ca@1692885234
• controller-manager
  Server certificate
  subject=CN = localhost@1692885230
  issuer=CN = localhost-ca@1692885230

whereas others are ok (api server, k3s server, kubelet). For example api server:
Server certificate
subject=CN = kube-apiserver
issuer=CN = personal-server-ca

I didn't see any specific cert config for these which are not using my CA.

[brandond]

Did you provision the cluster using Rancher, or just standalone using the k3s install script? Where (on which ports, or in which files) are you finding these certs? Can you provide the output of kubectl get node -o yaml ?

[sushiMix]

Used the k3s install script.
I used a simple curl -k -v to check the returned certificates:

• https://localhost:10257
• https://localhost:10258
• https://localhost:10259
  The same test for the API server on port 6444 is fine with the wanted CA.

[brandond]

Yeah, the health-check listener certs for the controller-manager, scheduler, and so on are just generated in-memory by the Kubernetes components themselves, and are not persisted to disk anywhere. That is, as far as I know, common behavior across most Kubernetes distros. You can see this occurring in the startup logs:

Sep 07 20:35:56 systemd-node-1 k3s[292]: I0907 20:35:56.781652     292 serving.go:355] Generated self-signed cert in-memory
Sep 07 20:35:59 systemd-node-1 k3s[292]: I0907 20:35:59.866262     292 serving.go:355] Generated self-signed cert in-memory
Sep 07 20:36:00 systemd-node-1 k3s[292]: I0907 20:36:00.056729     292 serving.go:355] Generated self-signed cert in-memory
Sep 07 20:36:00 systemd-node-1 k3s[292]: I0907 20:36:00.798396     292 serving.go:355] Generated self-signed cert in-memory

This is standard behavior when using either custom CA certs, or using the CAs generated by K3s.

[sushiMix]

Thanks to tell me that is health endpoint only (I didn't find it in the documentation) and some other health/metric endpoints are HTTP only