Multi-site setup with nodes behind ipsec-tunnels networking issue

[JoostVanDerSluis]

Hi all,

I have setup to play around with multi-site solutions.

The cluster has three control-plane nodes, at three different locations. Two on an on-premise location and one node running in an Azure VM (spot).

The on-premise sites are connected to the Azure-vnet through a vpn-gateway (ipsec). (fcos/4.15)

Everything seems to work, although slow (as expected) and the load (bandwidth) on the network is much higher that I expected.

But I discovered that there is a problem, which is hidden by the load-balancing system.

When I'm logged in on the console of the node in Azure (master2) I cannot make a direct call to the apiserver pod on another node (master1). But when I'm on the console of one of the other nodes (master0) I can make the same call and then it works.

The issue seems to be with the node in Azure, so behind the ipsec-tunnel.

Strange thing is, that ping works, that I get an immediate response when I connect to a wrong port. But the connection 'freezes' when I use the correct port and is disconnected after a timeout. So, when it is a firewall issue, the incoming connection goes well, but the answer is stuck. (While it is not a routing issue, as ping works)

Any ideas how to investigate this?

On local machine:

[joost@fed4k ~]$ kubectl get pods -n openshift-apiserver -o wide
NAME                         READY   STATUS    RESTARTS         AGE   IP            NODE                  NOMINATED NODE   READINESS GATES
apiserver-6756469f89-45dnp   2/2     Running   2                17d   10.130.0.49   master1.okd.cnoc.nl   <none>           <none>
apiserver-6756469f89-rwmlx   2/2     Running   26 (2d19h ago)   14d   10.129.0.4    master2               <none>           <none>
apiserver-6756469f89-w4mmh   2/2     Running   2                17d   10.128.0.78   master0.okd.cnoc.nl   <none>           <none>

On master0 node/console, attempt to connect to apiserver on master1

core@master0:~$ curl -k https://10.130.0.49:8443/readyz
okcore@master0:~$ 

On master2 node/console, attempt to connect to apiserver on master1, timeout

core@master2:~$ curl -k https://10.130.0.49:8443/readyz -v
* processing: https://10.130.0.49:8443/readyz
*   Trying 10.130.0.49:8443...
* Connected to 10.130.0.49 (10.130.0.49) port 8443
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* Recv failure: Connection reset by peer
* OpenSSL SSL_connect: Connection reset by peer in connection to 10.130.0.49:8443  
* Closing connection
curl: (35) Recv failure: Connection reset by peer

Invalid port, couldn't connect

core@master2:~$ curl -k https://10.130.0.49:8442/readyz -v
* processing: https://10.130.0.49:8442/readyz
*   Trying 10.130.0.49:8442...
* connect to 10.130.0.49 port 8442 failed: Connection refused
* Failed to connect to 10.130.0.49 port 8442 after 120 ms: Couldn't connect to server
* Closing connection
curl: (7) Failed to connect to 10.130.0.49 port 8442 after 120 ms: Couldn't connect to server

Ping succeeds

core@master2:~$ ping 10.130.0.49
PING 10.130.0.49 (10.130.0.49) 56(84) bytes of data.
64 bytes from 10.130.0.49: icmp_seq=1 ttl=62 time=32.1 ms
64 bytes from 10.130.0.49: icmp_seq=2 ttl=62 time=30.2 ms
^C
--- 10.130.0.49 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 30.167/31.132/32.098/0.965 ms
core@master2:~$