Artifacts: Workflow in different NS results in executor error: You need to configure artifact storage

[kalote]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

I'm configuring argo-workflows on multiple namespace:

• main argo-wf NS, which handles the argo-wf app (with server and controller)
• application backend NS, which runs wf

Argo-Workflows is deployed using argoCD + helm in its dedicated NS (argo-wf).

I added the following config (redacted to show the important bits):

argo-workflows:
  artifactRepositoryRef:
    artifact-repositories:
      annotations:
        workflows.argoproj.io/default-artifact-repository: default
      default:
        s3:
          bucket: argowf-artifacts-bucket
          endpoint: s3.amazonaws.com
          region: us-east-1
          keyFormat: "workflow\
                      /{{workflow.creationTimestamp.Y}}\
                      /{{workflow.creationTimestamp.m}}\
                      /{{workflow.creationTimestamp.d}}\
                      /{{workflow.name}}\
                      /{{pod.name}}"
          useSDKCreds: true

When I run jobs in the backend NS, it ends up failing with executor error: You need to configure artifact storage error.

My guess is that the pod (running in backend NS) doesn't find the CM with the artifactRepositoryRef configuration, as it is in the argo-wf NS, not in the backend NS.

Version(s)

v3.5.11

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: key-only-artifacts
spec:
  serviceAccountName: argo-exec
  entrypoint: main
  templates:
    - name: main
      dag:
        tasks:
          - name: generate
            template: generate
          - name: consume
            template: consume
            dependencies:
              - generate
    - name: generate
      container:
        image: busybox
        command: [sh, -c]
        args: ["echo hello world > /tmp/report.txt"]
      outputs:
        artifacts:
          - name: report
            path: /tmp/report.txt
            archive:
              none: {}
            s3:
              key: my-report
    - name: consume
      container:
        image: busybox
        command: [sh, -c]
        args: ["cat /tmp/report"]
      inputs:
        artifacts:
          - name: report
            path: /tmp/report
            s3:
              key: my-report

Logs from the workflow controller

At the start of the controller, I see this:

time="2024-10-28T17:43:13.407Z" level=info msg="Configuration:\nartifactRepository: {}\ninitialDelay: 0s\nmetricsConfig: {}\nnodeEvents:\n enabled: true\npodSpecLogStrategy: {}\nsso:\n clientId:\n key: \"\"\n clientSecret:\n key: \"\"\n issuer: \"\"\n redirectUrl: \"\"\n sessionExpiry: 0s\ntelemetryConfig: {}\n"

Why artifactRepository is empty?

No other logs are very relevant

Logs from in your workflow's wait container

Starting Workflow Executor
Using executor retry strategy
Executor initialized
Starting deadline monitor
Main container completed
No Script output reference in workflow. Capturing script output ignored
No output parameters
Saving output artifacts
Staging artifact: report
Copying /tmp/report.txt from container base image layer to /tmp/argo/outputs/artifacts/report.tgz
/var/run/argo/outputs/artifacts/tmp/report.txt.tgz -> /tmp/argo/outputs/artifacts/report.tgz
executor error: You need to configure artifact storage. More information on how to do this can be found in the docs: https://argo-workflows.readthedocs.io/en/release-3.5/configure-artifact-repository/
Alloc=7443 TotalAlloc=13524 Sys=23141 NumGC=4 Goroutines=8
You need to configure artifact storage. More information on how to do this can be found in the docs: https://argo-workflows.readthedocs.io/en/release-3.5/configure-artifact-repository/

[jswxstw]

My guess is that the pod (running in backend NS) doesn't find the CM with the artifactRepositoryRef configuration, as it is in the argo-wf NS, not in the backend NS.

You are right.
You should also add artifact-repositories configmap to backend namespace. You can also configure artifactRepository in workflow controller configmap if you do not want to add artifact-repositories configmap in every required namespace.

[kalote]

Thanks for your support 🙏

You should also add artifact-repositories configmap to backend namespace

Ok that's what I thought!

You can also configure artifactRepository in workflow controller configmap

Can you elaborate on this please? You mean, adding a artifactRepository entry in the workflow manifest? Do you have the link to the documentation of that please?
My goal is to factorize as much as possible, which means not adding the S3 config on each workflow if possible.

add artifact-repositories configmap in every required namespace.

So, just adding the configMap in the backend namespace is enough? Is there a way to indicate in which NS this CM should be created in the helm chart?

Thanks o/

[jswxstw]

Can you elaborate on this please? You mean, adding a artifactRepository entry in the workflow manifest? Do you have the link to the documentation of that please?

You can read this documentation: Workflow Controller Config Map.

My goal is to factorize as much as possible, which means not adding the S3 config on each workflow if possible.

No matter which method you use, artifactRepository will always be loaded into each workflow.

So, just adding the configMap in the backend namespace is enough?

Yes, add artifact-repositories ConfigMap or modify workflow controller ConfigMap, choose one.

Is there a way to indicate in which NS this CM should be created in the helm chart?

Sorry, I'm not familiar with Argo Helm.

[kalote]

Thanks for the complete & thorough explanation 🙏