fix(operator): allow retries to consider exit code from init container and don't consider node as pending if init failed. Fixes #11354/#10717/#10045

[tooptoop4]

Fixes #11354 and #10717 and #10045

Before this fix it would always go into pending because main container was waiting state (

argo-workflows/workflow/controller/operator.go

Line 1404 in 4742e9d

woc.markNodePhase(ctrNodeName, wfv1.NodePending)

) even though init container already terminated with non-0 exit

This supersedes #13852

cc @terrytangyuan

[jswxstw]

Before this fix it would always go into pending because main container was waiting state even though init container already terminated with non-0 exit

This will only be encountered when using ContainerSet, right?

[tooptoop4]

Before this fix it would always go into pending because main container was waiting state even though init container already terminated with non-0 exit

This will only be encountered when using ContainerSet, right?

no, see the logs/comments in the linked issues

[toralf]

We do regularly observe this issue here at work too for 3.5.12

[epifanov6]

We face this issue regularly, adding retry might very help.
We use 3.6.2.

[jswxstw]

Before this fix it would always go into pending because main container was waiting state even though init container already terminated with non-0 exit

@tooptoop4 I think your PR does not truly resolve this issue, because the nodes in the containerSet are initialized to Pending by default. Could you please add some tests?

argo-workflows/workflow/controller/container_set_template.go

Line 35 in 9d1d2cd

_ = woc.initializeNode(ctxNodeName, wfv1.NodeTypeContainer, templateScope, orgTmpl, node.ID, wfv1.NodePending, opts.nodeFlag)

[tooptoop4]

@jswxstw this has nothing to do with containerset, i don't run that workflow type

[jswxstw]

@jswxstw this has nothing to do with containerset, i don't run that workflow type

But the line of code you posted here is only relevant to the issue with containerSet.

argo-workflows/workflow/controller/operator.go

Line 1404 in 4742e9d

woc.markNodePhase(ctrNodeName, wfv1.NodePending)

Also, if you are not using containerSet, the node will be Failed if init container fails.
Could you provide the workflow you used if this issue does exist?

[tooptoop4]

that block is not limited to containerSet

[jswxstw]

that block is not limited to containerSet

Please provide the workflow you tested to prove it.

[jswxstw]

@toralf @epifanov6 Could you please clarify whether the issue you encountered is related to the missing exit code of the init container, or is it that the node is Pending when the init container fails?

[tooptoop4]

see linked issues for all the details

[epifanov6]

@jswxstw Init container is in state "Terminated", because of the reason "Error".
Then main container is in state "Waiting".
I believe that retry of the init container could help it this situation.

[jswxstw]

@jswxstw Init container is in state "Terminated", because of the reason "Error". Then main container is in state "Waiting". I believe that retry of the init container could help it this situation.

@epifanov6 The node should be Failed if init container fails, which by setting retryStrategy, you can achieve automatic retries.