CoreDNS is acting very strangely with zerotier + host-gw

[dzervas]

Environmental Info:
K3s Version: v1.27.4+k3s1

Node(s) CPU architecture, OS, and Version:

g0: 5.15.0-79-generic amd64
g1: 5.15.0-79-generic amd64
oracle0: 5.15.0-1040-oracle arm64 (oracle free tier)

Cluster Configuration:

3 servers

Describe the bug:

CoreDNS is acting extremely strangely. First of all, no error logs in its pod.

• sshing into the nodes and using nslookup <domain> 10.43.0.10 works - for internal and external domains
• Pods though that run on a node that coredns is not (e.g. coredns at g0 but pod in g1) can't reach it
• Can't resolve non-fqdn domains in any case - e.g. longhorn-backend or longhorn-backend.longhorn-system.svc don't resolve but longhorn-backend.longhorn-system.svc.cluster.local does

Steps To Reproduce:

This ain't easy but here goes:

• Install k3s in g0 and g1 normally (only --disable options)
• Set up zerotier
• "Migrate" the nodes to use zerotier (see bellow for the arguments)
• Add oracle0

k3s arguments: --secrets-encryption --disable=traefik,metrics-server --node-ip=<zt ip> --flannel-iface=<zt iface> --flannel-backend=host-gw --bind-address=<zt ip> --advertise-address=<zt ip> --tls-san=<zt ip>

Expected behavior:

CoreDNS to do its thing

Actual behavior:

CoreDNS behaving like a diva

I THINK restarting it fixes it for a bit

Additional context / logs:

I really have no idea where to check, where to start

HelmChart resource fails if the install pod doesn't land in the same node that coredns was deployed as it can't resolve the helm repo

[brandond]

Pods though that run on a node that coredns is not (e.g. coredns at g0 but pod in g1) can't reach it

UDP CNI traffic between nodes is being dropped. Are you sure that host-gw works properly over zerotier? Have you tried wireguard or wireguard+tailscale instead of zerotier+host-gw?

[dzervas]

no beacuse zerotier feels a bit too cozy to drop
The reason that I went for zerotier and not wireguard-native is because that the oracle0 instance is behind a NAT (oracle cloud does not give you your public IP in a NIC but does NAT)
and I'll set up nodes at home as well - so NAT is not negotiable

Maybe I can use VXLAN over zerotier?
Also are you talking about wireguard on the host or the flannel backend?
ChatGPT also gave me a note to lower the MTU (to e.g. 1400). would that help?

EDIT: Huh, just changed to VXLAN and this happened:

$ k run -i --tty --rm debug --image=busybox --restart=Never -- nslookup longhorn-backend && k delete pod debug
If you don't see a command prompt, try pressing enter.
warning: couldn't attach to pod/debug, falling back to streaming logs: unable to upgrade connection: container debug not found in pod debug_longhorn-system
Server:		10.43.0.10
Address:	10.43.0.10:53

Name:	longhorn-backend.longhorn-system.svc.cluster.local
Address: 10.43.117.40


** server can't find longhorn-backend.svc.cluster.local: NXDOMAIN

** server can't find longhorn-backend.svc.cluster.local: NXDOMAIN

** server can't find longhorn-backend.cluster.local: NXDOMAIN

** server can't find longhorn-backend.cluster.local: NXDOMAIN

Non-authoritative answer:

Non-authoritative answer:

EDIT 2: Nope, I just got lucky and the debug pod landed in the coredns node :/

[brandond]

You might try playing with the MTU, that could help if the drops are due to the packets being too large to pass through the tunnel. You might also search the issues and discussion for other reports of folks using zerotier to see if they have any reports of using it successfully, and if so what they had to change.

[brandond]

The reason that I went for zerotier and not wireguard-native is because that the oracle0 instance is behind a NAT (oracle cloud does not give you your public IP in a NIC but does NAT)
and I'll set up nodes at home as well - so NAT is not negotiable

Tailscale should work fine over NAT?
https://docs.k3s.io/installation/network-options#integration-with-the-tailscale-vpn-provider-experimental

[dzervas]

on the k3s cluster that I can't set up cause I don't have a VPN 😂

[dzervas]

as this feels more and more like "zerotier is not supported", I'm going with a custom wireguard approach. As in, set it up in mesh mode by hand and give it to flannel as it is to use VXLAN on top of it (am I thinking it correctly?)

[brandond]

I'm confused about the IDP signup issue; the available auth providers for tailscale are all dealbreakers for you?

Either way yeah, you might get zerotier to work, but we don't document it or support it so it'll probably require a fair bit of trial and error.

[dzervas]

the available auth providers for tailscale are all dealbreakers for you?
The problem isn't with only of the IDPs, it's the idea that I don't own my account - it's completely off-topic and has to do with my privacy/security compass and almost nothing to do with technical issues

in the case of using manually provisioned wireguard (since etcd + flannel wireguard backend don't go together) should I just go with the VXLAN backend?

(BTW: I'm sorry that I'm dragging you off-topic, I deeply appreciate your responses <3 )

[brandond]

etcd + flannel wireguard is fine as long as all the etcd nodes have direct connectivity to each other. You just can't (shouldn't) spread the etcd nodes across multiple locations regardless of VPN providers, in order to keep the latency down.

See https://github.com/flannel-io/flannel/blob/master/Documentation/backends.md for more info on the available flannel backends.