Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Inconsistent variable substitution in Inputs.Parameters block #13890

Closed
3 of 4 tasks
encigem opened this issue Nov 11, 2024 · 3 comments · Fixed by #13921
Closed
3 of 4 tasks

Inconsistent variable substitution in Inputs.Parameters block #13890

encigem opened this issue Nov 11, 2024 · 3 comments · Fixed by #13921
Assignees
@MasonM
Labels
area/templating Templating with `{{...}}` type/bug

Comments

@encigem
Copy link

encigem commented Nov 11, 2024

Pre-requisites

  • I have double-checked my configuration
  • I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
  • I have searched existing issues and could not find a match for this bug
  • I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

I'm having some issue with variable substitutions for 'inputs.paramters' blocks in Workflow files.
For example, i've provided a YAML file which reproduces my issue. In 'step1' the name: "{{workflow.parameters.cm-name-full}}" works within the configMapKeyRef , but with the name: "{{workflow.parameters.cm-name}}-cm" in step2 it does not.

screenshot_2024-11-08_at_16-30-12_substitute-test-fpg66___docklin-il___workflows_-_argo

What Happened?: The step with partial variable string input failed, but the step with input consisting only of a string with a variable succeeded.

What I expected?: I expected that both of the steps should've been able to substitute the given variable in the 'inputs.parameters' block

Version(s)

v3.5.10, latest(b26ed4a)

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: v1
kind: ConfigMap
data:
  testVar: testVal
metadata:
  labels:
    workflows.argoproj.io/configmap-type: Parameter
  name: test-cm
  namespace: argo

---

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: substitute-test-
  namespace: argo
spec:
  entrypoint: steps
  arguments:
    parameters:
      - name: image
        value: "alpine:3.17"
      - name: cm-key
        value: "testVar"
      - name: cm-name
        value: "test"
      - name: cm-name-full
        value: "test-cm"
  templates:
  - name: steps
    steps:
    - - name: step1
        template: step1
    - - name: step2
        template: step2

  - name: step1
    inputs:
      parameters:
        - name: cm-var
          valueFrom:
            configMapKeyRef:
              name: "{{workflow.parameters.cm-name-full}}" # only variable defined; succeeds
              key: "{{workflow.parameters.cm-key}}"
    container:
      image: "{{workflow.parameters.image}}"
      command: [sh, -c]
      args: ["echo 'testVar value is: {{inputs.parameters.cm-var}}'"]

  - name: step2
    inputs:
      parameters:
        - name: cm-var
          valueFrom:
            configMapKeyRef:
              name: "{{workflow.parameters.cm-name}}-cm"  # combine variable with string; fails
              key: "{{workflow.parameters.cm-key}}"
    container:
      image: "{{workflow.parameters.image}}"
      command: [sh, -c]
      args: ["echo 'testVar value is: {{inputs.parameters.cm-var}}'"]

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

time="2024-11-08T16:25:42.231Z" level=info msg="Processing workflow" Phase= ResourceVersion=725462700 namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.237Z" level=info msg="Task-result reconciliation" namespace=test-il numObjs=0 workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.237Z" level=info msg="Updated phase  -> Running" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.237Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.237Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.237Z" level=info msg="Steps node substitute-test-fpg66 initialized Running" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.238Z" level=info msg="StepGroup node substitute-test-fpg66-2823940808 initialized Running" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.238Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.238Z" level=info msg="Pod node substitute-test-fpg66-296026515 initialized Pending" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.264Z" level=info msg="Created pod: substitute-test-fpg66[0].step1 (substitute-test-fpg66-step1-296026515)" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.265Z" level=info msg="Workflow step group node substitute-test-fpg66-2823940808 not yet completed" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.265Z" level=info msg="TaskSet Reconciliation" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.265Z" level=info msg=reconcileAgentPod namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.286Z" level=info msg="Workflow update successful" namespace=test-il phase=Running resourceVersion=725462704 workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.266Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=725462704 namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.267Z" level=info msg="Task-result reconciliation" namespace=test-il numObjs=1 workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.267Z" level=info msg="task-result changed" namespace=test-il nodeID=substitute-test-fpg66-296026515 workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="node changed" namespace=test-il new.message= new.phase=Succeeded new.progress=0/1 nodeID=substitute-test-fpg66-296026515 old.message= old.phase=Pending old.progress=0/1 workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="Step group node substitute-test-fpg66-2823940808 successful" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="node substitute-test-fpg66-2823940808 phase Running -> Succeeded" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="node substitute-test-fpg66-2823940808 finished: 2024-11-08 16:25:52.268414392 +0000 UTC" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="StepGroup node substitute-test-fpg66-3964671805 initialized Running" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="SG Outbound nodes of substitute-test-fpg66-296026515 are [substitute-test-fpg66-296026515]" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="Skipped node substitute-test-fpg66-3148240723 initialized Error (message: unable to retrieve inputs.parameters.cm-var from ConfigMap: ConfigMap '{{workflow.parameters.cm-name}}-cm' does not exist. Please make sure it has the label workflows.argoproj.io/configmap-type: Parameter to be detectable by the controller)" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=error msg="Mark error node" error="step group deemed errored due to child substitute-test-fpg66[1].step2 error: unable to retrieve inputs.parameters.cm-var from ConfigMap: ConfigMap '{{workflow.parameters.cm-name}}-cm' does not exist. Please make sure it has the label workflows.argoproj.io/configmap-type: Parameter to be detectable by the controller" namespace=test-il nodeName="substitute-test-fpg66[1]" workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="node substitute-test-fpg66-3964671805 phase Running -> Error" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="node substitute-test-fpg66-3964671805 message: step group deemed errored due to child substitute-test-fpg66[1].step2 error: unable to retrieve inputs.parameters.cm-var from ConfigMap: ConfigMap '{{workflow.parameters.cm-name}}-cm' does not exist. Please make sure it has the label workflows.argoproj.io/configmap-type: Parameter to be detectable by the controller" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="node substitute-test-fpg66-3964671805 finished: 2024-11-08 16:25:52.268667688 +0000 UTC" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="step group substitute-test-fpg66-3964671805 was unsuccessful: step group deemed errored due to child substitute-test-fpg66[1].step2 error: unable to retrieve inputs.parameters.cm-var from ConfigMap: ConfigMap '{{workflow.parameters.cm-name}}-cm' does not exist. Please make sure it has the label workflows.argoproj.io/configmap-type: Parameter to be detectable by the controller" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="Outbound nodes of substitute-test-fpg66-3148240723 is [substitute-test-fpg66-3148240723]" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="Outbound nodes of substitute-test-fpg66 is [substitute-test-fpg66-3148240723]" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="node substitute-test-fpg66 phase Running -> Failed" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="node substitute-test-fpg66 message: step group deemed errored due to child substitute-test-fpg66[1].step2 error: unable to retrieve inputs.parameters.cm-var from ConfigMap: ConfigMap '{{workflow.parameters.cm-name}}-cm' does not exist. Please make sure it has the label workflows.argoproj.io/configmap-type: Parameter to be detectable by the controller" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="node substitute-test-fpg66 finished: 2024-11-08 16:25:52.268736598 +0000 UTC" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="TaskSet Reconciliation" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg=reconcileAgentPod namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="Updated phase Running -> Failed" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="Updated message  -> step group deemed errored due to child substitute-test-fpg66[1].step2 error: unable to retrieve inputs.parameters.cm-var from ConfigMap: ConfigMap '{{workflow.parameters.cm-name}}-cm' does not exist. Please make sure it has the label workflows.argoproj.io/configmap-type: Parameter to be detectable by the controller" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="Marking workflow completed" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.300Z" level=info msg="Workflow update successful" namespace=test-il phase=Failed resourceVersion=725462793 workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.302Z" level=info msg="Queueing Failed workflow test-il/substitute-test-fpg66 for delete in 168h0m0s due to TTL"
time="2024-11-08T16:26:02.382Z" level=info msg="Queueing Failed workflow test-il/substitute-test-fpg66 for delete in 167h59m50s due to TTL"
time="2024-11-08T16:46:02.378Z" level=info msg="Queueing Failed workflow test-il/substitute-test-fpg66 for delete in 167h39m50s due to TTL"
time="2024-11-08T17:06:02.389Z" level=info msg="Queueing Failed workflow test-il/substitute-test-fpg66 for delete in 167h19m50s due to TTL"
time="2024-11-08T17:25:52.321Z" level=info msg="cleaning up pod" action=deletePod key=test-il/substitute-test-fpg66-step1-296026515/deletePod
time="2024-11-08T17:26:02.383Z" level=info msg="Queueing Failed workflow test-il/substitute-test-fpg66 for delete in 166h59m50s due to TTL"

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

time="2024-11-11T13:15:40.526Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2024-11-11T13:15:40.526Z" level=info msg="No output parameters"
time="2024-11-11T13:15:40.526Z" level=info msg="No output artifacts"
time="2024-11-11T13:15:40.527Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: artifacts/2024/11/11/substitute-test-7bbdj/substitute-test-7bbdj-step1-1047580867/main.log"
time="2024-11-11T13:15:40.527Z" level=info msg="Creating minio client using static credentials" endpoint="seliis3obj.seli.gic.company.se:10444"
time="2024-11-11T13:15:40.527Z" level=info msg="Saving file to s3" bucket=testdev-moore030 endpoint="seliis3obj.seli.gic.company.se:10444" key=artifacts/2024/11/11/substitute-test-7bbdj/substitute-test-7bbdj-step1-1047580867/main.log path=/tmp/argo/outputs/logs/main.log
time="2024-11-11T13:15:40.601Z" level=info msg="Save artifact" artifactName=main-logs duration=74.237363ms error="<nil>" key=artifacts/2024/11/11/substitute-test-7bbdj/substitute-test-7bbdj-step1-1047580867/main.log
time="2024-11-11T13:15:40.601Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2024-11-11T13:15:40.601Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2024-11-11T13:15:40.632Z" level=info msg="Alloc=7710 TotalAlloc=17424 Sys=22357 NumGC=5 Goroutines=10"
@tooptoop4
Copy link
Contributor

this should be marked as feature not bug

@chengjoey
Copy link
Contributor 

      - name: cm-key
        value: "testVar"

Directly replace it with cm-var here, configMapKeyRef can solve your problem?

MasonM added a commit to MasonM/argo-workflows that referenced this issue Nov 19, 2024
@MasonM
fix: consistent variable substitution for configMapKeyRef. Fixes argo…
4ee0f0b 
…proj#13890

Support for subsituting variables in `configMapKeyRef` was added in
argoproj#7542 by manually parsing
templates out of the string instead of using the standard
`template.Replace()` functionality. Since the templates have the same
syntax as simple template tags (e.g. `{{workflow.parameters.cm-name}}`),
users reasonably expect it to work the same, but it doesn't, which leads
to confusion.

This replaces the manual parsing code with the standard
`template.Replace()` function that everything else uses.

Signed-off-by: Mason Malone <[email protected]>
@MasonM MasonM self-assigned this Nov 19, 2024
@MasonM MasonM mentioned this issue Nov 19, 2024
Merged
@MasonM MasonM added the area/templating Templating with `{{...}}` label Nov 19, 2024
@MasonM
Copy link
Contributor

MasonM commented Nov 19, 2024 •
edited
Loading

I agree that this constitutes a bug. I entered #13921 to fix it.

Thanks for the detailed bug report @encigem!

@tczhao tczhao closed this as completed in f2159dc Nov 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@MasonM MasonM

Labels
area/templating Templating with `{{...}}` type/bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: consistent variable substitution for configMapKeyRef. Fixes #13890 MasonM/argo-workflows
4 participants
@MasonM @chengjoey @tooptoop4 @encigem