v3.6.3 regression. Can't reliably call workflowTemplates using workflowTemplateRef

[tico24]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

1. Upgrade to 3.6.3
2. Have a simple workflowTemplate that has some parameters
3. Call that using templateRef and withItems
4. Error: failed to resolve {{item.path}}

Version(s)

v3.6.3

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: 363-test-tmp
  namespace: ci
spec:
  templates:
    - name: 363-test-tmp
      nodeSelector:
        nodegroup: arm-spot
      inputs:
        parameters:
          - name: path
          - name: service
          - name: build_arg
          - name: run_on
          - name: arch
          - name: docker_org
      container:
        image: alpine
        command:
          - sh
          - -c
          - |
            echo "path: {{inputs.parameters.path}}"
            echo "service: {{inputs.parameters.service}}"
            echo "build_arg: {{inputs.parameters.build_arg}}"
            echo "run_on: {{inputs.parameters.run_on}}"
            echo "arch: {{inputs.parameters.arch}}"
            echo "docker_org: {{inputs.parameters.docker_org}}"
---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: 363-test-
  namespace: ci
spec:
  activeDeadlineSeconds: 10800
  entrypoint: main
  serviceAccountName: ci

  templates:
    - name: main
      dag:
        tasks:
          - name: withitems
            templateRef:
              name: 363-test-tmp
              template: 363-test-tmp
            arguments:
              parameters:
                - name: path
                  value: "{{item.path}}"
                - name: service
                  value: "{{item.service}}"
                - name: build_arg
                  value: "{{item.arg}}"
                - name: run_on
                  value: "{{item.run_on}}"
                - name: arch
                  value: "{{item.arch}}"
                - name: docker_org
                  value: "{{item.docker_org}}"
            withItems:
              - {
                  path: "services",
                  service: "id",
                  arg: "",
                  run_on: "arm-spot",
                  arch: "arm64",
                  docker_org: "pipekit13",
                }
              - {
                  path: "services",
                  service: "events-handler",
                  arg: "",
                  run_on: "arm-spot",
                  arch: "arm64",
                  docker_org: "pipekit13",
                }

Logs from the workflow controller

Nothing relevant. The workflow does not start.

Logs from in your workflow's wait container

n/a

[tico24]

This also fails (referencing the same workflowTemplate as above):

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: 363-test-
  namespace: ci
spec:
  activeDeadlineSeconds: 10800
  entrypoint: main
  serviceAccountName: ci

  templates:
    - name: main
      dag:
        tasks:
          - name: pre
            template: pre
          - name: outputstest
            templateRef:
              name: 363-test-tmp
              template: 363-test-tmp
            arguments:
              parameters:
                - name: path
                  value: "{{tasks.pre.outputs.parameters.pre}}"
                - name: service
                  value: "foo"
                - name: build_arg
                  value: "foo"
                - name: run_on
                  value: "foo"
                - name: arch
                  value: "foo"
                - name: docker_org
                  value: "foo"
            depends: pre

    - name: pre
      container:
        image: alpine
        command:
          - sh
          - -c
          - |
            echo "pre" > /tmp/pre
      outputs:
        parameters:
          - name: pre
            valueFrom:
              path: /tmp/pre

[Joibel]

This is caused by #14053.

I'll be reverting this on main and release-3.6 and doing a v3.6.4 release to fix it.