outputs from steps not passed as parameters to templateRef

[dberardo-com]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

i am using templateRef to call one workflow inside another "master" template.

the master template produces some output in intermediate steps which i would like to pass as parameter to the child workflow.

the params are passed correctly to the templateRef node, but fail to get interpolated in steps below:

Version(s)

v3.5.1

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

# inside the master template i am using this step

        - - name: setup-mosquitto
            arguments:
              parameters:
                - name: tenant_name
                  value: '{{workflow.parameters.tenant_name}}'
                - name: tenant_namespace
                  value: >-
                    {{steps.create-namespace.outputs.parameters.tenant_namespace}}


# in the child template i am using simple workflow params:

spec:
  templates:
    - name: main
      inputs:
        parameters:
          - name: tenant_name
          - name: tenant_namespace
          - name: tenant_id
      outputs: {}
      metadata: {}
      dag:
        tasks:
          - name: ...
            template: ...
            arguments:
              parameters:
                - name: tenant_name
                  value: majico-tree-{{workflow.parameters.tenant_name}}
                - name: namespace
                  value: '{{workflow.parameters.tenant_namespace}}'

Logs from the workflow controller

no need

Logs from in your workflow's wait container

no need

[tczhao]

Hi @dberardo-com

you have a template main that specifies dag, input...

• workflow.parameters will refer to the argument field of a running workflow:

kind: Workflow
spec:
  arguments:
    parameters:
      - name: foo
        value: bar

• if you want to refer to the main.inputs in template main's dag task you need to refer to it as inputs.parameters:

spec:
  templates:
    - name: main
      inputs:
        parameters:
          - name: tenant_name
          - name: tenant_namespace
          - name: tenant_id
      outputs: {}
      metadata: {}
      dag:
        tasks:
          - name: ...
            template: ...
            arguments:
              parameters:
                - name: tenant_name
                  value: majico-tree-{{inputs.parameters.tenant_name}}
                - name: namespace
                  value: '{{inputs.parameters.tenant_namespace}}'

see example
https://github.com/argoproj/argo-workflows/blob/734b5b6e9f18ebe385df3e41117e67a1e8764939/examples/arguments-parameters.yaml

[dberardo-com]

i will look into your answer but i am not sure this is in line with the problem i have. basically the issue is that i am defining both workflow.parameters and outputs in the master template but then i can only read the workflow.parameters in the child template.

if seems like the outputs values generated in the master template are not propagated in the children. does it make sense?

is there a way to generate an output in the master and read it in the children ?

btw when i say children i dont mean tasks or steps, i mean child templates, meaning referenced templates

[agilgur5]

[...] We must be able to run the workflow

You didn't provide a complete example, so it's hard to illustrate, but Tianchu's examples above are correct.

is there a way to generate an output in the master and read it in the children ?

That is what inputs are for, per above.

[dberardo-com]

ok, after some work i was able to setup a reproducible example to test a bunch of cases and you can see that there are some issues with it:

here the parent workflow:

metadata:
  name: test-parent
spec:
  templates:
    - name: main
      inputs:
        parameters:
          - name: parent_message
      outputs: {}
      metadata: {}
      steps:
        - - name: step1
            inline:
              inputs: {}
              outputs:
                parameters:
                  - name: output_message
                    valueFrom:
                      path: /out
              metadata: {}
              container:
                name: main
                image: argoproj/argosay:v2
                command:
                  - /bin/sh
                  - '-c'
                args:
                  - >-
                    echo {{workflow.parameters.parent_message}} > /out && cat
                    /out
                resources: {}
            arguments: {}
        - - name: step2
            arguments:
              parameters:
                - name: message
                  value: '{{workflow.parameters.parent_message}}'
                - name: message2
                  value: '{{steps.step1.outputs.parameters.output_message}}'
                - name: message_extra1
                  value: '{{workflow.parameters.parent_message}}'
                - name: message_extra2
                  value: '{{steps.step1.outputs.parameters.output_message}}'
            templateRef:
              name: test-sub
              template: main
  entrypoint: main
  arguments:
    parameters:
      - name: parent_message
        value: hi from parent
  ttlStrategy:
    secondsAfterCompletion: 300
  workflowMetadata:
    labels:
      example: 'true'

here the child

metadata:
  name: test-sub
spec:
  templates:
    - name: main
      inputs:
        parameters:
          - name: message
          - name: message2
      outputs: {}
      metadata: {}
      steps:
        - - name: step1
            template: step1
            arguments:
              parameters:
                - name: message
                  value: '{{workflow.parameters.message}}'
                - name: message2
                  value: '{{workflow.parameters.message2}}'
                - name: message3
                  value: '{{entrypoints.main.inputs.parameters.message}}'
    - name: step1
      inputs:
        parameters:
          - name: message
            default: empty1
          - name: message2
            default: empty2
          - name: message3
            default: empty3
      outputs: {}
      metadata: {}
      container:
        name: main
        image: argoproj/argosay:v2
        command:
          - /bin/sh
          - '-c'
        args:
          - >-
            echo - {{workflow.parameters.message}} -
            {{inputs.parameters.message}} - {{workflow.parameters.message2}} -
            {{inputs.parameters.message2}} - {{inputs.parameters.message3}}
        resources: {}
  entrypoint: main
  arguments:
    parameters:
      - name: message
        value: hello argo 1
      - name: message2
        value: hello argo 2
  ttlStrategy:
    secondsAfterCompletion: 300
  workflowMetadata:
    labels:
      example: 'true'

so again, my aim is to pass both workflow parameters AND steps output from the parent flow to the child flow

[tczhao]

Thanks for the example.
The idea is the spec.arguments.parameters.x is available as workflow.parameters.x to the entire workflow but spec.arguments.parameters.x is reading from the workflow you submitted

e.g.
when you submit test-sub
the spec.arguments.parameters.x in the test-sub is available to the entire workflow

but when you submit test-parent, the spec.arguments.parameters.x from test-parent is available to the entire workflow
and spec.arguments.parameters.x from test-sub are ignored.

Your options are

1. having template only use parameter inputs.parameters.x, and set workflow.parameters.x as default value for the input parameter
2. or use global output parameter https://github.com/argoproj/argo-workflows/blob/main/examples/global-outputs.yaml

For 1. see below example

metadata:
  name: test-sub
spec:
  templates:
    - name: main
      inputs:
        parameters:
          - name: message
            value: '{{workflow.parameters.message}}'
          - name: message2
            value: '{{workflow.parameters.message2}}'
      outputs: {}
      metadata: {}
      steps:
        - - name: step1
            template: step1
            arguments:
              parameters:
                - name: message
                  value: '{{inputs.parameters.message}}'
                - name: message2
                  value: '{{inputs.parameters.message2}}'
    - name: step1
      inputs:
        parameters:
          - name: message
            default: empty1
          - name: message2
            default: empty2
      outputs: {}
      metadata: {}
      container:
        name: main
        image: argoproj/argosay:v2
        command:
          - /bin/sh
          - '-c'
        args:
          - >-
            echo - 
            {{inputs.parameters.message}}
            {{inputs.parameters.message2}}
        resources: {}
  entrypoint: main
  arguments:
    parameters:
      - name: message
        value: hello argo 1
      - name: message2
        value: hello argo 2
  ttlStrategy:
    secondsAfterCompletion: 300
  workflowMetadata:
    labels:
      example: 'true'

[dberardo-com]

i think i got it:

• all templates (steps) should use inputs.* inside arguments.* instead of workflow.*
• the main.inputs.* should use workflow.*
• (important!) the memoize cache keys should use workflow.*
• how about conditionals "when" ?? should i use workflow.* or inputs.* there?

i think this is the gist of it!

oh dear, this deserves some dedicated piece documentation IMO.

[dberardo-com]

@AleCeli fyi

[dberardo-com]

@tczhao @agilgur5 thanks, i did make the needed modifications and indeed it all worked out.

i am still wondering however how to handle:

• default values --> in the child workflow i simply use default value in workflow args, but since default values cannot be applied to input params (AFAIK) i wonder if one must always re-define all default values as workflow params in the parent workflow as well ... this was my solution but is really annoying to maintain ...
• caching keys and conditionals --> can those fields (and other "meta-fields" for the matter) use input params that are passed through by the parent workflow? i kind of had problems handling this use case

cheers

[tczhao]

We may need a case by case example to understand the exact problem.

Overall, I think you could have a look at global parameters https://github.com/argoproj/argo-workflows/blob/main/examples/global-outputs.yaml
e.g. have first step to generate parameter use by the entire workflow, and all subsequent step can use them when needed without explicitly defining inputs all the way from parent to child workflow