[🐛 Bug]: Unexpected token '<', "<!DOCTYPE "... is not valid JSON unable to connect websocket on port forwarding and via Go Teleport

[sevaho]

Describe the bug

I am testing out 0.1.42 via helm (deployed Via ArgoCD) and I am unable to load Providers or do anything that relies on websockets.

I am getting the error via these ways:

• When I do port-forward via Kubernetes kubectl port-forward services/keep-frontend 3000 I am unable to see Providers.
• When proxying via Go Teleport I also get the same errors. Go teleport makes it possible to proxy traffic through https with an OIDC in front. In essence it calls the internal Service in Kuberentes (keep-frontend:3000).

I think I need to adjust some values in the helm chart, but I am not certain which ones, anyone could help?

To Reproduce
Steps to reproduce the behavior:

1. Go to https://keep.teleport.asic.li/providers
2. See error

Screenshots
If applicable, add screenshots to help explain your problem.

[dosubot[bot]]

Hello @sevaho! I'm here to help you with any bugs, questions, or contributions you may have. Let's work together to solve the issue you're facing.

To address the WebSocket connection issues you're experiencing with Keep when using port-forwarding and Go Teleport, you might need to adjust some configurations related to WebSocket handling in the Helm chart. Here are some key configurations to check:

1. Pusher Configuration: Ensure that the Pusher settings are correctly configured. These include:

   • PUSHER_DISABLED: Should be set to "false" to enable WebSocket connections.
   • PUSHER_HOST: Ensure this is set to the correct hostname.
   • PUSHER_PORT: Verify that this is set to the correct port, typically 6001.
   • PUSHER_APP_KEY, PUSHER_APP_ID, and PUSHER_APP_SECRET: Ensure these are correctly set for authentication.

2. Ingress Configuration: Make sure the ingress is correctly routing WebSocket traffic. The default path for WebSocket connections is /websocket, and this should be correctly configured in your ingress settings.

3. SSL/TLS Settings: If you're using SSL/TLS, ensure that PUSHER_USE_SSL is set to "true" if required.

4. Proxy Settings: Since you're using Go Teleport, ensure that it is correctly configured to handle WebSocket traffic. This might involve setting up appropriate headers or ensuring that the WebSocket path is correctly proxied.

These configurations are crucial for establishing and maintaining WebSocket connections in Keep. Adjusting these settings in your Helm chart should help resolve the connection issues you're facing ^[1][2].

To continue talking to Dosu, mention @dosu.

[shahargl]

hey @sevaho, I'll be happy to help.

what do you see on your devtools network panel? can you add a screen shot of the URL that Keep tries to get?

[sevaho]

[sevaho]

When port-forwarding I get this:

I assume it is missing the port?

[shahargl]

@sevaho yea thanks! the wss errors are unrelated with the backend errors.

With our default helm chart what happens is that all Keep services (backend, frontend and websocket) are served behind one ingress:

1. /v2/* -> backend
2. /websocket -> websocket for live updates
3. /* -> all other routes are going to the UI

the helm chart support both haproxy/nginx as ingress that knows how to rewrite those requests.

it looks like (1) and (2) doesn't go through. so my suspect is that you don't have ingress (either nginx or haproxy) on your kubernetes cluster. may it be the case?

[talboren]

@sevaho have you been able to solve this problem? happy to invite you to join our Slack community: https://slack.keephq.dev - @shahargl and I can probably help you more easily there :)

[sevaho]

I'll join the slack, to dig deeper into this! I understand the problem now, I need to create a custom nginx config so that I can route internal traffic in my Kube cluster.

[sevaho]

Thanks a lot @talboren for hopping on a call and helping me out!!

The solution was to move the nginx rewrite inside a location block:

        location ~ ^/v2(/|$)(.*) {
            rewrite ^/v2/(.*)$ /$1 break; # <--------
            proxy_pass http://keep-backend:8080;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }

Thus for anyone seeing this issue, and wanting to deploy keephq/keep behind a 'single' nginx instance so it can be used when doing kubectl port-forward or when you are using 'Go Teleport' here is the full nginx deployment:

apiVersion: v1
kind: ConfigMap
metadata:
  name: keep-nginx-config
data:
  default.conf: |
    server {
        listen 80;
        server_name _;

        proxy_read_timeout 3600s;
        proxy_send_timeout 3600s;
        proxy_connect_timeout 3600s;
        proxy_buffering off;
        proxy_http_version 1.1;
        http2 off;

        location ^~ /websocket {
          rewrite ^/websocket(/|$)(.*) /$2 break;
          proxy_pass http://keep-websocket:6001;
          proxy_http_version 1.1;
          proxy_set_header Upgrade $http_upgrade;
          proxy_set_header Connection "upgrade";
          proxy_set_header Host $host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto $scheme;
          proxy_set_header X-Forwarded-Host $http_host;
          proxy_cache_bypass $http_upgrade;
        }

        location ~ ^/websocket(/|$)(.*) {
            rewrite ^/v2/(.*)$ /$1 break; # Remove /v2 from the path
            proxy_pass http://keep-websocket:6001;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }

        location ~ ^/v2(/|$)(.*) {
            rewrite ^/v2/(.*)$ /$1 break; # Remove /v2 from the path
            proxy_pass http://keep-backend:8080;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }

        location ~ ^/(.*) {
            proxy_pass http://keep-frontend:3000;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: keep-nginx
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
        volumeMounts:
          - name: keep-nginx-config
            mountPath: /etc/nginx/conf.d
      volumes:
        - name: keep-nginx-config
          configMap:
            name: keep-nginx-config
            items:
              - key: default.conf
                path: default.conf
---
apiVersion: v1
kind: Service
metadata:
  name: keep-nginx
spec:
  type: NodePort
  selector:
    app: nginx
  ports:
    - port: 80
      targetPort: 80