[QUESTION/HELP] Cannot establish websocket connections with pods in k3d-deployed clusters

[gbates101]

Question / Where do you need Help?

Unsure if this is a bug or user error, but my problem is twofold:

• When creating a cluster with k3d-proxy loadbalancer enabled, my application cannot establish a websocket connection (HTTP 400) to apps running in-cluster (all other routing works properly).
• When attempting to create a cluster without a loadbalancer, I cannot define a port (for a NodePort service or otherwise) without receiving this error (and mapping no ports means I can't expose services):

gbates@tensorbook:~/Projects/anno $ k3d cluster create anno --config k3d.yaml
INFO[0000] Using config file k3d.yaml (k3d.io/v1alpha4#simple) 
FATA[0000] failed to transform ports: port-mapping of type 'proxy' specified, but loadbalancer is disabled

Scope of your Question

Is your question related to a specific version of k3d (or k3s)?

Not to my knowledge. This is my first use of k3d (coming from minikube), so I've only ever used this version.

Version:

k3d version v5.3.0
k3s version v1.22.6-k3s1 (default)

Additional Info

I've tailed the servicelb container logs, all traefik pod logs, and our websocket application pod logs, but see no activity when visiting wss://localhost:8080/connect. Visiting http://localhost:8080/connect shows an access log entry in our websocket app logs, which tells me that ingress routing is working appropriately for "normal" HTTP traffic, and all other routes defined in the ingress "work" as expected (200, 404, or otherwise). I've used both traefik and nginx as ingress controllers, both with the same outcome (all HTTP traffic routes appropriately, and websocket connections do not). I wanted to rule out the loadbalancer as the point of failure, but I can't start a cluster with exposed services without one due to aforementioned issue at the beginning of this post. We have several identical installations of our app running behind loadbalancers in EKS, and also versions running locally in minikube, all of which can establish websocket connections with our websocket app just fine.

k3d.yaml without loadbalancer:

apiVersion: k3d.io/v1alpha4
kind: Simple
metadata:
  name: anno
servers: 1
agents: 3
kubeAPI:
  hostIP: "0.0.0.0"
  hostPort: "6445"
volumes:
  - volume: /opt/k3s:/var/lib/rancher/k3s/storage@all
    nodeFilters:
      - server:0
      - agent:*
ports:
  - port: 8080:32080
    nodeFilters:
      - agent:*
  # Use these for nginx ingress controller (if using)
  # - port: 8080:32080
  #   nodeFilters:
  #     - server:*
  # - port: 8081:32081
  #   nodeFilters:
  #     - server:*
registries:
  create:
    name: registry.localhost
    host: "0.0.0.0"
    hostPort: "5000"
options:
  k3d:
    timeout: "60s"
    disableLoadbalancer: true
    # loadbalancer:
    #   configOverrides:
    #     - settings.workerConnections=2048
  # Use this to disable traefik
  # k3s:
  #   extraArgs: # additional arguments passed to the `k3s server|agent` command; same as `--k3s-arg`
  #     - arg: --disable=traefik
  #       nodeFilters:
  #         - server:*

k3d.yaml with loadbalancer enabled:

apiVersion: k3d.io/v1alpha4
kind: Simple
metadata:
  name: anno
servers: 1
agents: 3
kubeAPI:
  hostIP: "0.0.0.0"
  hostPort: "6445"
volumes:
  - volume: /opt/k3s:/var/lib/rancher/k3s/storage@all
    nodeFilters:
      - server:0
      - agent:*
ports:
  - port: 8080:80
    nodeFilters:
      - server:*
  # Use these for nginx ingress controller (if using)
  # - port: 8080:32080
  #   nodeFilters:
  #     - server:*
  # - port: 8081:32081
  #   nodeFilters:
  #     - server:*
registries:
  create:
    name: registry.localhost
    host: "0.0.0.0"
    hostPort: "5000"
options:
  k3d:
    timeout: "60s"
    loadbalancer:
      configOverrides:
        - settings.workerConnections=2048
  # Use this to disable traefik
  # k3s:
  #   extraArgs: # additional arguments passed to the `k3s server|agent` command; same as `--k3s-arg`
  #     - arg: --disable=traefik
  #       nodeFilters:
  #         - server:*

Ingress resource definition:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: anno-engine
  annotations:    
    ingress.kubernetes.io/ssl-redirect: "false"
  labels:    
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/instance: anno
    app.kubernetes.io/part-of: engine
    helm.sh/chart: engine-1.0.0
spec:
  rules:
    - http:
        paths:
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: anno-engine-api
                port:
                  number: 9000
          - path: /docs
            pathType: Prefix
            backend:
              service:
                name: anno-engine-api
                port:
                  number: 9000
          - path: /openapi.json
            pathType: Prefix
            backend:
              service:
                name: anno-engine-api
                port:
                  number: 9000
          - path: /files
            pathType: Prefix
            backend:
              service:
                name: anno-tusd
                port:
                  number: 80       
          - path: /auth
            pathType: Prefix
            backend:
              service:
                name: anno-keycloak-http
                port:
                  number: 80
          - path: /
            pathType: Prefix
            backend:
              service:
                name: anno-engine-ui
                port:
                  number: 3000
          - path: /connect
            pathType: Prefix
            backend:
              service:
                name: anno-engine-websocket-service
                port:
                  number: 9000

If there's any other info you need I'm happy to provide it.

[iwilltry42]

Hi @gbates101 , thanks for opening this issue!
Here's the solution to the actual k3d issue part:

When attempting to create a cluster without a loadbalancer, I cannot define a port (for a NodePort service or otherwise) without receiving this error (and mapping no ports means I can't expose services):

ports:
  - port: 8080:32080
    nodeFilters:
      - agent:0:direct # direct: do not go via k3d-proxy (doesn't exist anyway now), 0: because you cannot direct map the same port to multiple containers

For the other part: k3d-proxy does a simple proxy_pass and nothing else, so I wonder how it could be blocking here, but you can check the logs inside 👍

[gbates101]

Ah, thanks for the config snippet. Will give it a go. I'll verify my websocket app is working without the k3d-proxy in place and report back.

Regarding the k3d-proxy with websocet connections, I've checked the access logs for the k3d-proxy, traefik/nginx ingress controller, and the websocket app itself and there is zero log activity when attempting to upgrade a connection. Access log entries appear for normal HTTP traffic. The reason I think it's the k3d-proxy is because, coupled with the fact that websocket upgrade requests return 400: Connection refused, the nginx.conf requires the Upgrade and Connection headers to be set explicitly:

NGINX supports WebSocket by allowing a tunnel to be set up between a client and a backend server. For NGINX to send the Upgrade request from the client to the backend server, the Upgrade and Connection headers must be set explicitly, as in this example:

location /wsapp/ {
    proxy_pass http://wsbackend;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_set_header Host $host;
}

I suspected I intended to modify the k3d-proxy's nginx config to add a location block for the /connect route with these headers set explicitly as an experiment, but it appears k3d-proxy was implemented using the stream module, which doesn't support location blocks and, presumably, doesn't support explicitly setting headers either (I wasn't able to find any supporting document or get a working config, hence the attempt to disable it altogether).

[gbates101]

Alright, after further investigation it's definitely user error. Turns out there's an important distinction between ws:// and wss:// when operating locally without TLS 🤦... Once I configured our frontend to use ws:// things work normally.

Appreciate your time. I'll try this setup with the k3d-proxy enabled to confirm, but based on the root cause I'm guessing it'll work fine.

[iwilltry42]

Great that you figured it out! 👍
I missed the wss:// part as well from your comment 🤷‍♂️
I was wondering how k3d-proxy could interfere there, as it doesn't operate on that level (as you saw, it's only plain tcp proxying via stream block).
Good luck going forward with this :)