argo-workflows controller fails to process jobs if ClusterWorkflowTemplate CRD is not present

[dmerrick]

Describe the bug

If you set clusterWorkflowTemplates.enabled=false, the corresponding CRD will not get installed

The issue is, the argo-workflows controller will still try to look for that CRD on startup, which leads to a controller failure with a confusingly-terse error message:

W1115 18:02:15.645866       1 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *unstructured.Unstructured: the server could not find the requested resource
E1115 18:02:15.645922       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource

To make matters worse, the controller does not gracefully progress beyond this error. It will continue to attempt to fetch ClusterWorkflowTemplate CRD objects instead of moving on to process workflows. This leaves all workflows stuck in a pending state.

To get around this, you can set clusterWorkflowTemplates.enabled=true and apply the chart, which will create the corresponding CRD. You can then revert that change and the controller will function as expected.

Related helm chart

argo-workflows

Helm chart version

0.42.7

To Reproduce

On a new install:

1. Set controller.clusterWorkflowTemplates.enabled=false and crds.install=true
2. Apply the helm chart
3. Launch a workflow

Expected behavior

I would expect that the controller would be able to function without the ClusterWorkflowTemplate CRD if clusterWorkflowTemplates.enabled=false

Screenshots

No response

Additional context

No response

[yu-croco]

According to the condition in CRD, I wonder .Values.server.clusterWorkflowTemplates.enabled = false will help you?
Ref: https://github.com/argoproj/argo-helm/blob/main/charts/argo-workflows/templates/crds/argoproj.io_clusterworkflowtemplates.yaml#L1-L2

[dmerrick]

Upon reflection, I believe this is a bug in argo-workflows itself and not necessarily the helm chart. I could see two fixes:

1. update argo-workflows controller process to take a flag that disables clusterworkflowtemplate CRD lookups
2. update this helm chart to create the clusterworkflowtemplate CRD even if it's set to be disabled

[yu-croco]

I believe this is a bug in argo-workflows itself and not necessarily the helm chart.

Please open an issue in upstream since argo-helm focuses on providing Helm Chart, instead of handling the specific behavior of Argo.