Pod creation failed: unsupported VolumeMount option: subPath: tmp-dir-argo

[repl-thomas-lafreniere]

Trying a simple setup with quick start and running a basic cowsay workflow that triggers from an Argo Events github eventsource/sensor. The workflow argo executor fails to create the pod saying subPath is not an option on VolumeMount... This is on OCI OKE virtual node pool cluster, kubernetes 1.31.

apiVersion: v1
items:
- apiVersion: v1
  kind: Pod
  metadata:
    annotations:
      kubectl.kubernetes.io/default-container: main
      workflows.argoproj.io/node-id: github-wj946
      workflows.argoproj.io/node-name: github-wj946
    creationTimestamp: "2025-02-25T01:10:49Z"
    labels:
      workflows.argoproj.io/completed: "false"
      workflows.argoproj.io/workflow: github-wj946
    name: github-wj946
    namespace: argo-events
    ownerReferences:
    - apiVersion: argoproj.io/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: Workflow
      name: github-wj946
      uid: 695ca746-26aa-47a1-9718-05c27bad89b4
    resourceVersion: "1502022"
    uid: 73349469-f7b8-43e7-9f12-d65a0a4ddfda
  spec:
    containers:
    - command:
      - argoexec
      - wait
      - --loglevel
      - info
      - --log-format
      - text
      - --gloglevel
      - "0"
      env:
      - name: ARGO_POD_NAME
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.name
      - name: ARGO_POD_UID
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.uid
      - name: ARGO_WORKFLOW_NAME
        value: github-wj946
      - name: ARGO_WORKFLOW_UID
        value: 695ca746-26aa-47a1-9718-05c27bad89b4
      - name: ARGO_CONTAINER_NAME
        value: wait
      - name: ARGO_NODE_ID
        value: github-wj946
      - name: ARGO_INCLUDE_SCRIPT_OUTPUT
        value: "false"
      - name: ARGO_DEADLINE
        value: "0001-01-01T00:00:00Z"
      - name: ARGO_PROGRESS_FILE
        value: /var/run/argo/progress
      - name: ARGO_PROGRESS_PATCH_TICK_DURATION
        value: 1m0s
      - name: ARGO_PROGRESS_FILE_TICK_DURATION
        value: 3s
      image: quay.io/argoproj/argoexec:v3.6.4
      imagePullPolicy: IfNotPresent
      name: wait
      resources:
        requests:
          cpu: 10m
          memory: 64Mi
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /tmp
        name: tmp-dir-argo
        subPath: "0"
      - mountPath: /var/run/argo
        name: var-run-argo
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-s7xjp
        readOnly: true
    - args:
      - Update README.md
      command:
      - /var/run/argo/argoexec
      - emissary
      - --loglevel
      - info
      - --log-format
      - text
      - --gloglevel
      - "0"
      - --
      - cowsay
      env:
      - name: ARGO_CONTAINER_NAME
        value: main
      - name: ARGO_NODE_ID
        value: github-wj946
      - name: ARGO_INCLUDE_SCRIPT_OUTPUT
        value: "false"
      - name: ARGO_DEADLINE
        value: "0001-01-01T00:00:00Z"
      - name: ARGO_PROGRESS_FILE
        value: /var/run/argo/progress
      - name: ARGO_PROGRESS_PATCH_TICK_DURATION
        value: 1m0s
      - name: ARGO_PROGRESS_FILE_TICK_DURATION
        value: 3s
      image: docker/whalesay:latest
      imagePullPolicy: Always
      name: main
      resources: {}
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /var/run/argo
        name: var-run-argo
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-s7xjp
        readOnly: true
    dnsPolicy: ClusterFirst
    enableServiceLinks: true
    initContainers:
    - command:
      - argoexec
      - init
      - --loglevel
      - info
      - --log-format
      - text
      - --gloglevel
      - "0"
      env:
      - name: ARGO_POD_NAME
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.name
      - name: ARGO_POD_UID
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.uid
      - name: ARGO_WORKFLOW_NAME
        value: github-wj946
      - name: ARGO_WORKFLOW_UID
        value: 695ca746-26aa-47a1-9718-05c27bad89b4
      - name: ARGO_CONTAINER_NAME
        value: init
      - name: ARGO_TEMPLATE
        value: '{"name":"whalesay","inputs":{},"outputs":{},"metadata":{},"container":{"name":"","image":"docker/whalesay:latest","command":["cowsay"],"args":["Update
          README.md"],"resources":{}}}'
      - name: ARGO_NODE_ID
        value: github-wj946
      - name: ARGO_INCLUDE_SCRIPT_OUTPUT
        value: "false"
      - name: ARGO_DEADLINE
        value: "0001-01-01T00:00:00Z"
      - name: ARGO_PROGRESS_FILE
        value: /var/run/argo/progress
      - name: ARGO_PROGRESS_PATCH_TICK_DURATION
        value: 1m0s
      - name: ARGO_PROGRESS_FILE_TICK_DURATION
        value: 3s
      image: quay.io/argoproj/argoexec:v3.6.4
      imagePullPolicy: IfNotPresent
      name: init
      resources:
        requests:
          cpu: 10m
          memory: 64Mi
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /var/run/argo
        name: var-run-argo
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-s7xjp
        readOnly: true
    nodeName: 10.0.10.3
    preemptionPolicy: PreemptLowerPriority
    priority: 0
    restartPolicy: Never
    schedulerName: default-scheduler
    securityContext: {}
    serviceAccount: default
    serviceAccountName: default
    terminationGracePeriodSeconds: 30
    tolerations:
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    volumes:
    - emptyDir: {}
      name: var-run-argo
    - emptyDir: {}
      name: tmp-dir-argo
    - name: kube-api-access-s7xjp
      projected:
        defaultMode: 420
        sources:
        - serviceAccountToken:
            expirationSeconds: 3607
            path: token
        - configMap:
            items:
            - key: ca.crt
              path: ca.crt
            name: kube-root-ca.crt
        - downwardAPI:
            items:
            - fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
              path: namespace
  status:
    conditions:
    - lastProbeTime: null
      lastTransitionTime: "2025-02-25T01:10:50Z"
      status: "True"
      type: PodScheduled
    - lastProbeTime: null
      lastTransitionTime: null
      reason: ContainerCreateFailed
      status: "False"
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: null
      reason: ContainerCreateFailed
      status: "False"
      type: Ready
    message: 'Pod creation failed: unsupported VolumeMount option: subPath: tmp-dir-argo'
    phase: Pending
    qosClass: Burstable
    reason: ContainerCreateFailed
kind: List
metadata:
  resourceVersion: ""

[repl-thomas-lafreniere]

Ok seems like a limitation of Oracle virtual nodes: "Maximum of 1 volume of type emptyDir can currently be defined in the pod spec."
https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengcomparingvirtualwithmanagednodes_topic.htm

Is there a way to consolidate the run volume and the tmp volume?

volumes:
    - emptyDir: {}
      name: var-run-argo
    - emptyDir: {}
      name: tmp-dir-argo

[repl-thomas-lafreniere]

yes. The issue is the virtual nodes restriction on OCI specifically. Closing this as answered.