Unable to connect to the server / Client.Timeout inside vcluster

[jwedel]

What happened?

Hi,

I am experiencing timeouts with kubectl commands when bing inside a vcluster:

vcluster connect vcluster -n <namespace> --insecure
$ vcluster connect vcluster -n <namespace> --insecure
info   Using vcluster vcluster load balancer endpoint: 20.23.32.81
done √ Switched active kube context to vcluster_vcluster_pq-2_pq-aks
- Use `vcluster disconnect` to return to your previous kube context
- Use `kubectl get namespaces` to access the vcluster
$ kubectl get namespaces
Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

The get namespace command takes a long time to return with this error message. Also other kubectl commands fail in the same way.

Outside of the vcluster, kubectl works as expected.

What did you expect to happen?

kubectl should work also inside the vcluster and e.g. return the namespaces or pods.

How can we reproduce it (as minimally and precisely as possible)?

No idea how to reproduce it. :(

Anything else we need to know?

I'm running on a company mac with local http proxy that forwards to an authenticated proxy. Proxy env vars are set in shell.

Host cluster Kubernetes version

$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.2", GitCommit:"5835544ca568b757a8ecae5c153f317e5736700e", GitTreeState:"clean", BuildDate:"2022-09-21T14:25:45Z", GoVersion:"go1.19.1", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.8", GitCommit:"83d00b7cbf10e530d1d4b2403f22413220c37621", GitTreeState:"clean", BuildDate:"2022-11-09T19:50:11Z", GoVersion:"go1.17.11", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.25) and server (1.23) exceeds the supported minor version skew of +/-1

Host cluster Kubernetes distribution

# Write here

vlcuster version

$ vcluster --version
vcluster version 0.13.0

Vcluster Kubernetes distribution(k3s(default)), k8s, k0s)

aks (k8s)

OS and Arch

OS: MacOS 13.1
Arch: ARM (Apple M1 Pro)

[FabianKramm]

Hey @jwedel, thanks for creating this issue! This usually means the loadbalancer is not working correctly, does it work when you use port-forwarding through the Kubernetes api server instead?

[jwedel]

@FabianKramm I think I found the issue. The server url configured in kube config when being inside the vcluster is not reachable from inside our VPN. I am currently trying to get this solved. I will come back it is actually fixed.

[alexandradragodan]

Heyaa, @jwedel 👋

Have you managed to get past the VPN and have the server reachable?
Waiting on your confirmation before closing this issue.

Thanks!
Have a great day ahead 💯

[jwedel]

Hi, that’s very kind! Unfortunately not. I’ve created a ticket with our corporate IT. They told me that are currently working on it. I have no idea how long this will take. 😤

[jwedel]

Hey @alexandradragodan , our it support is still "working" on the problem, they have found some potentially related issues:

• https://jackiechen.blog/2018/12/05/kubectl-unable-to-connect-minikube-on-vpn/
• Can't use Minikube on VPN kubernetes/minikube#1099 (comment)

I know, they don't have anything to do with vcluster BUT they use the same VPN client (Cisco) and the get the same error on kubectl.

Their scenario is using kubectl -> k3s (locally). They solved it by changing the configuration of k3s to add a port forwarding to somehow circumvent the VPN client.

My question would be: Is there a way to do the same for vcluster as we are not allowed to change anything in the VPN client config.

[matskiv]

Hello @jwedel
you could try using the --server=https://127.0.0.1:8443 flag for the vcluster connect command to change the address of the server used in the kube context, but I don't quite get how they created the local proxy.

You can also try creating the vcluster with a different config. Seems like you are exposing it via LoadBalancer, the alternatives are documented here + another alternative is to ignore all the options in that doc and then vcluster connect will use Kubernetes port-forwarding feature to connect. To help more with this we would need to see the command that you use to create the vcluster and any helm values that are supplied.

[matskiv]

I will close this issue because of the inactivity. We can reopen it if the problem persists and additional information is provided.