CLI ignores TLS client certificate with --argo-http1 and --argo-server

[notniknot]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

We have a corporate proxy that requires client certificate authentication to access services.

Therefore, I specify --client-certificate and --client-key

This does work:
argo --server https://kubernetes-endpoint -n argo --token='XXX' --client-certificate ~/.secrets/client.pem --client-key ~/.secrets/key.pem list

This does not work:
argo --argo-http1 --client-certificate ~/.secrets/client.pem --client-key ~/.secrets/key.pem --loglevel DEBUG --argo-server argo-server-instance:443 -n argo --token=XXX list

Output:

DEBU[2024-08-05T22:10:42.749Z] CLI version                                   version="{v3.5.10 2024-08-01T05:52:04Z 25829927431d9a0f46d17b72ae74aedb8d700884 v3.5.10 clean go1.21.12 gc darwin/amd64}"
DEBU[2024-08-05T22:10:42.749Z] Client options                                opts="(argoServerOpts=(url=argo-server-instance:443,path=,secure=true,insecureSkipVerify=false,http=true),instanceID=)"
DEBU[2024-08-05T22:10:42.751Z]                                               listOpts="&ListOptions{LabelSelector:,FieldSelector:,Watch:false,ResourceVersion:,TimeoutSeconds:nil,Limit:0,Continue:,AllowWatchBookmarks:false,ResourceVersionMatch:,}"
DEBU[2024-08-05T22:10:42.753Z] curl -X GET -H 'Authorization: ******' -d '' 'https://argo-server-instance:443/api/v1/workflows/argo?fields=metadata%2Citems.metadata%2Citems.spec%2Citems.status.phase%2Citems.status.message%2Citems.status.finishedAt%2Citems.status.startedAt%2Citems.status.estimatedDuration%2Citems.status.progress' 
FATA[2024-08-05T22:10:42.830Z] Get "https://argo-server-instance:443/api/v1/workflows/argo?fields=metadata%2Citems.metadata%2Citems.spec%2Citems.status.phase%2Citems.status.message%2Citems.status.finishedAt%2Citems.status.startedAt%2Citems.status.estimatedDuration%2Citems.status.progress": EOF

Whereas when I copy the curl command from the output, fill in the token and add the certs, it works again:
curl --cert ~/.secrets/client.pem --key ~/.secrets/key.pem -X GET -H 'Authorization: Bearer XXX' -d '' 'https://argo-server-instance:443/api/v1/workflows/argo?fields=metadata%2Citems.metadata%2Citems.spec%2Citems.status.phase%2Citems.status.message%2Citems.status.finishedAt%2Citems.status.startedAt%2Citems.status.estimatedDuration%2Citems.status.progress'

I suspect that the client certs are ignored when using --argo-http1 and --argo-server.

Version(s)

v3.5.10

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

Has nothing to do with workflows

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

[agilgur5]

If you're using the CLI's server mode, the CLI sends a request to the Server and needs a token to access it (i.e. the Server is running with --auth-mode=client), and then the Server sends a request to k8s. The provided SA bearer token is used and not a client cert in that case.

You could potentially open a feature request for Server support for client certs, but they don't correspond to a namespace and SA, so it might be hard to retrieve some features for those (e.g. the UI's default namespace; see also #13372)

EDIT: I screwed this up, see below

[notniknot]

@agilgur5 Thanks for your response. The issue here is, that we have a reverse proxy in front of the argo server. I don't want argo to use my client cert, I need to authenticate against the reverse proxy in order to reach the argo server. And somehow, with those args I cannot get through.

[agilgur5]

Oh. I totally missed the proxy part of your issue, sorry about that and thanks for elaborating. And you also clearly have a bearer token now that I read your cURL command more carefully

Also I misremembered what that flag is for, it's a TLS client cert, not a k8s one. Sorry I was just working with k8s auth and must've assumed that's what it was.

I dropped the bag on this one, sorry!

Let me reopen this then for investigation. There might be a reason it's ignored or it might just be an oversight or was only implemented for one mode.

[agilgur5]

I took a quick look since the CLI is pretty small. It looks like my initial thought process was partly correct actually, that flag is inherited from kubectl and so only used when connecting to k8s directly via client-go (the k8s Go SDK).

The CLI doesn't use that flag itself and just passes it through to client-go. In server mode, client-go is not used though.

So I guess this would be more of a feature request to reuse that flag in server mode as well. In other words to pass that flag to the gRPC/HTTP client as well

[notniknot]

Thanks for the investigation! I will create a feature request.

[agilgur5]

We can just re-use this issue as a feature request. It's somewhere in between a bug and a feature as I can understand why you would be confused that the flag doesn't always work since it doesn't mention it's used only for k8s mode.

IIRC a lot of CLIs do this exact thing, which is confusing too to be fair. Argo's is maybe a bit more confusing as it has multiple modes (the others I can remember are k8s only, so even if the inheritance isn't mentioned, it's at least still used the same).
metrics-server I remember having quite confusing flags as it's a server itself but also a k8s client but also is a server that k8s itself needs certs to (mTLS) and so disambiguation becomes quite complex 🫠 EDIT: ah yes me from 4 years ago: kubernetes-sigs/metrics-server#610