kubeadm unable to select an IP from default routes.

[wenzong]

What keywords did you search in kubeadm issues before filing this one?

Similar issues found, but all of them are closed.

#1156
#2323
kubernetes/kubernetes#95425
kubernetes/kubernetes#95779

BUG REPORT

Versions

• kubeadm version:
  • &version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:40:11Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• kubectl version:
  • Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:41:28Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}
  • Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:35:25Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}
• Cloud provider or hardware configuration:
  • Virtual Machine in company
• OS (e.g. from /etc/os-release): CentOS Linux release 7.3.1611 (Core)
• Kernel: 3.10.0-514.el7.x86_64
• Others:

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         127.0.0.53      0.0.0.0         UG    0      0        0 tunnat
10.0.0.0        10.176.0.1      255.0.0.0       UG    0      0        0 eth0
10.176.0.0      0.0.0.0         255.255.0.0     U     0      0        0 eth0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.19.0.0      0.0.0.0         255.255.255.0   U     0      0        0 cni0
172.19.1.0      172.19.1.0      255.255.255.0   UG    0      0        0 flannel.1
172.19.2.0      172.19.2.0      255.255.255.0   UG    0      0        0 flannel.1
172.19.3.0      172.19.3.0      255.255.255.0   UG    0      0        0 flannel.1
172.19.4.0      172.19.4.0      255.255.255.0   UG    0      0        0 flannel.1
172.19.5.0      172.19.5.0      255.255.255.0   UG    0      0        0 flannel.1
172.24.0.0      10.176.0.1      255.248.0.0     UG    0      0        0 eth0
192.168.0.0     10.176.0.1      255.255.0.0     UG    0      0        0 eth0

# ifconfig tunnat
tunnat: flags=209<UP,POINTOPOINT,RUNNING,NOARP>  mtu 1480
        inet 127.0.0.53  netmask 255.255.255.255  destination 127.0.0.53
        tunnel   txqueuelen 1  (IPIP Tunnel)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 25950  bytes 1884510 (1.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

# ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.176.238.179  netmask 255.255.0.0  broadcast 10.176.255.255
        ether 02:00:0a:b0:ee:b3  txqueuelen 1000  (Ethernet)
        RX packets 19895594  bytes 1992976835 (1.8 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8905218  bytes 1587626127 (1.4 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

What happened?

kubeadm join/config command not working properly.

# kubeadm config images list
could not convert cfg to an internal cfg: unable to select an IP from default routes.
To see the stack trace of this error execute with --v=5 or higher

# kubeadm config images list -v=5
I1113 16:24:47.911691  692692 interface.go:431] Looking for default routes with IPv4 addresses
I1113 16:24:47.911934  692692 interface.go:436] Default route transits interface "tunnat"
I1113 16:24:47.912330  692692 interface.go:208] Interface tunnat is up
I1113 16:24:47.912491  692692 interface.go:256] Interface "tunnat" has 1 addresses :[127.0.0.53/32].
I1113 16:24:47.912595  692692 interface.go:223] Checking addr  127.0.0.53/32.
I1113 16:24:47.912660  692692 interface.go:233] Non-global unicast address found 127.0.0.53
I1113 16:24:47.912991  692692 interface.go:208] Interface lo is up
I1113 16:24:47.913108  692692 interface.go:285] Interface "lo" has 1 addresses :[127.0.0.1/8].
I1113 16:24:47.913190  692692 interface.go:223] Checking addr  127.0.0.1/8.
I1113 16:24:47.913260  692692 interface.go:233] Non-global unicast address found 127.0.0.1
I1113 16:24:47.913326  692692 interface.go:208] Interface eth0 is up
I1113 16:24:47.913399  692692 interface.go:208] Interface tunl0 is up
I1113 16:24:47.913476  692692 interface.go:208] Interface tunnat is up
I1113 16:24:47.913540  692692 interface.go:208] Interface tunl1 is up
I1113 16:24:47.913599  692692 interface.go:208] Interface tunl2 is up
I1113 16:24:47.913674  692692 interface.go:208] Interface docker0 is up
I1113 16:24:47.913738  692692 interface.go:208] Interface cni0 is up
I1113 16:24:47.913808  692692 interface.go:208] Interface veth0d8cdca5 is up
I1113 16:24:47.913877  692692 interface.go:208] Interface veth77ceecc5 is up
I1113 16:24:47.913950  692692 interface.go:208] Interface flannel.1 is up
I1113 16:24:47.914011  692692 interface.go:431] Looking for default routes with IPv6 addresses
I1113 16:24:47.914076  692692 interface.go:458] No active IP found by looking at default routes
unable to select an IP from default routes.
could not convert cfg to an internal cfg

after delete the default route table, kubeadm works properly

# route del default
# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.0.0.0        10.176.0.1      255.0.0.0       UG    0      0        0 eth0
10.176.0.0      0.0.0.0         255.255.0.0     U     0      0        0 eth0
... ignore

# kubeadm config images list
W1113 16:36:06.504985  698012 common.go:149] WARNING: could not obtain a bind address for the API Server: no default routes found in "/proc/net/route" or "/proc/net/ipv6_route"; using: 0.0.0.0
W1113 16:36:07.030890  698012 version.go:103] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get "https://dl.k8s.io/release/stable-1.txt": dial tcp [2600:1901:0:26f3::]:443: connect: network is unreachable
W1113 16:36:07.031115  698012 version.go:104] falling back to the local client version: v1.22.3
k8s.gcr.io/kube-apiserver:v1.22.3
k8s.gcr.io/kube-controller-manager:v1.22.3
k8s.gcr.io/kube-scheduler:v1.22.3
k8s.gcr.io/kube-proxy:v1.22.3
k8s.gcr.io/pause:3.5
k8s.gcr.io/etcd:3.5.0-0
k8s.gcr.io/coredns/coredns:v1.8.4

tunnat is the device for outside Internet access, thus I have to pre-download all images over Internet and delete the tunnat route, then I can run kubeadm join and other subcommands.

What you expected to happen?

kubeadm should work on such routing table, or give us some other option to set the IP manually.

How to reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

[neolit123]

Hi. All kubernetes component IPs can be configured with the kubeadm API and with patches.
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/control-plane-flags/

If you do not pass the custom IPs all of them use the same default route detection and they will fail in your case. That is how k8s is designed, so instead of setting the IP per every component best to make the default route IP detection available on the host.

/kind support

[wenzong]

Hi. All kubernetes component IPs can be configured with the kubeadm API and with patches. https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/control-plane-flags/

If you do not pass the custom IPs all of them use the same default route detection and they will fail in your case. That is how k8s is designed, so instead of setting the IP per every component best to make the default route IP detection available on the host.

/kind support

Thanks for your reply.

But it's confusing since run kubeadm init with --apiserver-advertise-address won't complain such problem. However run kubeadm join with --apiserver-adverise-address will fail, and kubeadm config images list should have nothing to do with network settings, I guess?

[neolit123]

Most kubeadm commands dynamically populate a config in memory and IPs are part of the config.

The whole feature "configure all components with custom IPs" is already possible...But it is very messy for the user. Instead you can use the 'ip route' command to create a route that goes to any IP that you want..This is what i would like to see documented in the docs.

And contributions for that are welcome.

[wenzong]

Most kubeadm commands dynamically populate a config in memory and IPs are part of the config.

The whole feature "configure all components with custom IPs" is already possible...But it is very messy for the user. Instead you can use the 'ip route' command to create a route that goes to any IP that you want..This is what i would like to see documented in the docs.

And contributions for that are welcome.

Thanks again.

So use the patches or change the routing table, both should fix the complain.

Actually I used route command to fix the route and bypass the problem, what I am really looking for is the patches method. Will take some time to look into it.

Closing the issue for now.

[rijuchatterjee]

Most kubeadm commands dynamically populate a config in memory and IPs are part of the config.
The whole feature "configure all components with custom IPs" is already possible...But it is very messy for the user. Instead you can use the 'ip route' command to create a route that goes to any IP that you want..This is what i would like to see documented in the docs.
And contributions for that are welcome.

Thanks again.

So use the patches or change the routing table, both should fix the complain.

Actually I used route command to fix the route and bypass the problem, what I am really looking for is the patches method. Will take some time to look into it.

Closing the issue for now.

Can you provide the steps to fix the route issue and bypass the problem?

[wenzong]

Most kubeadm commands dynamically populate a config in memory and IPs are part of the config.
The whole feature "configure all components with custom IPs" is already possible...But it is very messy for the user. Instead you can use the 'ip route' command to create a route that goes to any IP that you want..This is what i would like to see documented in the docs.
And contributions for that are welcome.

Thanks again.
So use the patches or change the routing table, both should fix the complain.
Actually I used route command to fix the route and bypass the problem, what I am really looking for is the patches method. Will take some time to look into it.
Closing the issue for now.

Can you provide the steps to fix the route issue and bypass the problem?

I already post the command and route table above.

BTW, I'm not sure my workaround is suitable for your situation, please evaluate it then use at your own risk.(it may breaks your machine's network)

route del default

Recover command(work for me only)

route add default gw 127.0.0.53 tunnat

@rijuchatterjee

[double12gzh]

two methods:

1. add a new default gw with command route add default gw {gateway} via {device_name}
2. setup param --bind-address with the interface, which has ipv4 or ipv6 ip

[neolit123]

WIP docs to direct users to setting up default routes and not using the IP flags in k8s components, unless really needed:
kubernetes/website#43872

xref #2604 (comment)