[BUG] handle potential uncaught OOMKilled terminations

[pvditt]

Tracking issue

#4704

Why are the changes needed?

Propeller doesn't always fail OOMKilled pods as OOMKilled.

What changes were proposed in this pull request?

Catch potential OOMKilled errors that have a ExitCode: 0 + enrich error messages to mark error as OOMKilled.

Will follow up with a PR to enrich other error messages.

How was this patch tested?

• added unit test
• ran workflow

Setup process

import subprocess

from flytekit import Resources, task, workflow
from flytekit.core.workflow import WorkflowFailurePolicy


@task(requests=Resources(mem="500Mi"))
def subproc_oom():
    try:
        subprocess.run(["tail", "/dev/zero"], check=True)
    except subprocess.CalledProcessError as e:
        print(f"Return code {e.returncode}")
        raise e


@task(requests=Resources(mem="500Mi"))
def other_oom():
    l = []
    while True:
        l.append('a' * 10**6)


@workflow(
    failure_policy=WorkflowFailurePolicy.FAIL_AFTER_EXECUTABLE_NODES_COMPLETE
)
def workflow():
    subproc_oom()
    other_oom()

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Docs link

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (abef3f5) 58.54% compared to head (87dbc09) 58.56%.
Report is 7 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4793      +/-   ##
==========================================
+ Coverage   58.54%   58.56%   +0.01%     
==========================================
  Files         625      624       -1     
  Lines       53669    53600      -69     
==========================================
- Hits        31423    31391      -32     
+ Misses      19731    19692      -39     
- Partials     2515     2517       +2     

Flag	Coverage Δ
unittests	58.56% <100.00%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[EngHabu]

Nice!