Kubernetes DNS resolution not working on a new cluster

[biased-badger]

Problem

Kubernetes DNS resolution doesn't seem to be working in my freshly created cluster.

I was trying to install MetalLB on a freshly created Talos cluster via helm. The chart itself installed just fine, however adding IPAddressPool resource failed with the following error:

Error from server (InternalError): error when creating "metallb-ingress-pool.yaml": Internal error occurred: failed calling webhook "ipaddresspoolvalidationwebhook.metallb.io": failed to call webhook: Post "https://metallb-webhook-service.metallb-system.svc:443/validate-metallb-io-v1beta1-ipaddresspool?timeout=10s": context deadline exceeded

Trying to debug it, I followed this guide and I could see that none of my requests end up in the CoreDNS pods.

Context

I'm trying to create a cluster on VMware VSphere using latest (1.9.1) Talos distribution. I have all nodes in their own subnet and they are all using my internal DNS servers specified in machine/network/nameservers.

I tried different configurations for machine/features/hostDNS but none seem to affect this issue.

I also tried adding 10.96.0.10 (kube-dns) to machine/kubelet/clusterDNS but that also didn't help.

It feels that I'm doing something wrong here or I'm missing a critical detail about how that's supposed to work, so I'll grateful for any help!

Configs

Here are the patches that I apply to the default config:

- op: add
  path: /machine/network
  value:
    hostname: ${hostname}
    nameservers: ${my_dns_servers}
    interfaces:
      - interface: eth0
        addresses:
          - ${address}
        routes:
          - network: 0.0.0.0/0
            gateway: ${gateway}
---
- op: add
  path: /machine/time
  value:
    servers: ${my_dns_servers}

[smira]

If you're on VMWare, and using a CNI with packet encapsulation (like default Flannel), VMWare default vmxnet driver has issues that it doesn't handle it properly.

A usual workaround is to use Intel NIC emulation on VMWare side.

So probably CNI doesn't work which leads to CoreDNS being unavailable. You don't need to change any defaults to make it working, it should work out of the box.

[biased-badger]

Thank you! I spent quite a lot of time trying to solve this issue.

I was told by our IT department that we can't add an Intel NIC and we're expected to move to OpenStack soon, so I won't be able to confirm that it works with an Intel NIC.

[smira]

OpenStack has no issues like that, it's specific to VMWare environment.