Server behind Istio Gateway

[esn89]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

Downloaded, installed Argo Workflows 3.5.10, with secure=false and BASE_HREF=/ behind Istio, gives errors of:

upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection failure

Or, it just loads a blank page

Version(s)

3.5.10

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

No workflows yet being that I wasn't even able to get to that part

What I can tell you is that, I have set:

--secure false with BASE_HREF=/.

On Istio Gateway:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: dev-gateway
spec:
  selector:
    istio: dev-ingressgateway
  servers:
  - hosts:
    - workflows.mycompany.com
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      credentialName: istio-ingress-server-tls
      mode: SIMPLE

VirtualService:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: argo-vs
  namespace: argo
spec:
  gateways:
  - default/dev-gateway
  hosts:
  - workflows.mycompany.com
  http:
  - match:
    - uri:
        prefix: /
    rewrite:
      uri: /
    route:
    - destination:
        host: my-argo-workflows-server
        port:
          number: 2746

Kubernetes Service:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: server
    app.kubernetes.io/component: server
    app.kubernetes.io/instance: my-argo-workflows
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: argo-workflows-server
    app.kubernetes.io/part-of: argo-workflows
    app.kubernetes.io/version: v3.5.10
    helm.sh/chart: argo-workflows-0.41.14
  name: my-argo-workflows-server
  namespace: argo
spec:
  clusterIP: 172.16.141.165
  clusterIPs:
  - 172.16.141.165
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - port: 2746
    protocol: TCP
    targetPort: 2746
  type: ClusterIP
status:
  loadBalancer: {}

Logs from the workflow controller

I haven't ran any workflows yet

Logs from in your workflow's wait container

I haven't ran any workflows yet

I also took a look and did some searching and saw something similar here: #3080 (comment)
from @arnoldrw and @agilgur5 , I tried that but to no avail.

[agilgur5]

You filed this as a bug, but this is not an Argo bug. The Server works and has E2E tests on CI, is used during dev, and many people have it working, including in the Ingress examples you linked.

You also had it working yourself on Slack.
So this is an Ingress/Istio GW/VS configuration issue and not an Argo bug.

As the "Create a new issue" chooser says, issues are for bugs, while Slack and Discussions are for support. You already have a Slack thread and did receive some debugging help from the community already.