DNS over TLS not working

[manuelbuil]

Environmental Info:
RKE2 Version:

RKE2 February release: v1.29.2, v1.28.7, v1.27.11, v1.26.14

Node(s) CPU architecture, OS, and Version:

Cluster Configuration:

Describe the bug:

When using a TLS config in coredns, we get an error in the logs of coredns:

x509: certificate signed by unknown authority

The reason is that the path ls /etc/ssl/certs is empty and it should contain the typical CA certificates.

Steps To Reproduce:

• Installed RKE2:
  1 - Deploy RKE2 (e.g. v1.29.2+rke2r1)
  2 - Check the logs on coredns: (e.g. kubectl logs rke2-coredns-rke2-coredns-6fd7bb5597-4b5kn -n kube-system)

.:53
[INFO] plugin/reload: Running configuration SHA512 = c18591e7950724fe7f26bd172b7e98b6d72581b4a8fc4e5fc4cfd08229eea58f4ad043c9fd3dbd1110a11499c4aa3164cdd63ca0dd5ee59651d61756c4f671b7
CoreDNS-1.11.1
linux/amd64, go1.20.7 X:boringcrypto, ae2bbc29

3 - kubectl edit cm rke2-coredns-rke2-coredns -n kube-system
4 - Replace forward . /etc/resolv.conf\n by forward . tls://1.1.1.1:853 tls://1.0.0.1:853\n
5 - Consume the new config: kubectl rollout restart deployment rke2-coredns-rke2-coredns -n kube-system
6 - Wait for pods to come up. Once they are up, check the logs. You should see the problem:

.:53
[INFO] plugin/reload: Running configuration SHA512 = 6ac0e5f74fb935b974b697f3549b0d8c483c18fbf722f98f4c788c5b6059e96c4f9b7ed2f647a941bad8c7906563fefda900099aa7fcaa8756ea1fbcb6a8af5c
CoreDNS-1.11.1
linux/amd64, go1.20.7 X:boringcrypto, ae2bbc29
[ERROR] plugin/errors: 2 5749506477736180542.6615920629555885813. HINFO: tls: failed to verify certificate: x509: certificate signed by unknown authority

Expected behavior:

No error

Actual behavior:

Error which makes it impossible to reach DNS servers using TLS

Additional context / logs:

[mdrahman-suse]

Validated on master branch with commit c7cd05b

Environment/Config

Ubuntu 22.04

$ cat /etc/rancher/rke2/config.yaml
write-kubeconfig-mode: 644
token: summerheat
node-name: server1
debug: true

Testing steps

Thanks to Manuel for the detailed steps
Followed the steps mentioned here: #5550 (comment)

Replication:

$ rke2 -v
rke2 version v1.29.2+rke2r1 (08699dfffdf75a61a5e6064f9f8efe8ddae857fe)
go version go1.21.7 X:boringcrypto

• Before editing coredns configmap

kube-system   pod/rke2-coredns-rke2-coredns-6fd7bb5597-79njd              0/1     Running

$ k logs -n kube-system pod/rke2-coredns-rke2-coredns-6fd7bb5597-79njd
.:53
[INFO] plugin/reload: Running configuration SHA512 = c18591e7950724fe7f26bd172b7e98b6d72581b4a8fc4e5fc4cfd08229eea58f4ad043c9fd3dbd1110a11499c4aa3164cdd63ca0dd5ee59651d61756c4f671b7
CoreDNS-1.11.1
linux/amd64, go1.20.7 X:boringcrypto, ae2bbc29

• After editing coredns configmap and restarting coredns pod

kube-system   pod/rke2-coredns-rke2-coredns-699bb8cbc-6jbb2               1/1     Running 

$ k logs -n kube-system pod/rke2-coredns-rke2-coredns-699bb8cbc-6jbb2
.:53
[INFO] plugin/reload: Running configuration SHA512 = 6ac0e5f74fb935b974b697f3549b0d8c483c18fbf722f98f4c788c5b6059e96c4f9b7ed2f647a941bad8c7906563fefda900099aa7fcaa8756ea1fbcb6a8af5c
CoreDNS-1.11.1
linux/amd64, go1.20.7 X:boringcrypto, ae2bbc29
[ERROR] plugin/errors: 2 2168130729565867384.4141158716131606011. HINFO: tls: failed to verify certificate: x509: certificate signed by unknown authority

Validation:

$ rke2 -v
rke2 version v1.29.2+dev.c7cd05bf (c7cd05bf547712250bd7a47db69258dbf823c80a)
go version go1.21.7 X:boringcrypto

• Before editing coredns configmap

kube-system   pod/rke2-coredns-rke2-coredns-5f76c4f8b4-kdjvg             0/1     Running 

$ k logs -n kube-system pod/rke2-coredns-rke2-coredns-5f76c4f8b4-kdjvg
.:53
[INFO] plugin/reload: Running configuration SHA512 = c18591e7950724fe7f26bd172b7e98b6d72581b4a8fc4e5fc4cfd08229eea58f4ad043c9fd3dbd1110a11499c4aa3164cdd63ca0dd5ee59651d61756c4f671b7
CoreDNS-1.11.1
linux/amd64, go1.20.14 X:boringcrypto, ae2bbc29

• After editing coredns configmap and restarting coredns pod

kube-system   rke2-coredns-rke2-coredns-7c74f87bcc-jwkmt             1/1     Running

$ k logs -n kube-system pod/rke2-coredns-rke2-coredns-7c74f87bcc-jwkmt
.:53
[INFO] plugin/reload: Running configuration SHA512 = 6ac0e5f74fb935b974b697f3549b0d8c483c18fbf722f98f4c788c5b6059e96c4f9b7ed2f647a941bad8c7906563fefda900099aa7fcaa8756ea1fbcb6a8af5c
CoreDNS-1.11.1
linux/amd64, go1.20.14 X:boringcrypto, ae2bbc29