[CI] Fix smoke tests

[NickLucche]

smoke-test.sh currently throws cuda OOM errors due to forced shutdown of vLLM, which results in leaving cuda memory allocated. We wait for a gracious shutdown and use a helper script to print out stats about gpu memory, in case similar issues arise again.

[dtrifiro]

Looks great. Let's see if CI passes 😅

[NickLucche]

Looks great. Let's see if CI passes 😅

Yeah I wish..

�[36mINFO�[0m[2025-01-22T15:53:45Z] Build vllm-build-main-cuda-amd64 failed, printing logs: 
unable to retrieve container logs for cri-o://3991e41e9b2c5fddb955c7af723fb20e8c721587f0474d8abdc0ec7126316da1unable to retrieve container logs for cri-o://e4f0136398891296951e81f65771b4eac8399f5f411b7ec284cc41398dd701d7unable to retrieve container logs for cri-o://37e59768b199c3d88e31c1d16cc90920dbc6008812f614bfb39c45eb6f89b51e�[36mINFO�[0m[2025-01-22T15:53:45Z] Build vllm-build-main-cuda-amd64 previously failed from an infrastructure error (BuildPodEvicted), retrying... 

Other jobs are still pending though, I thought the smoke-test was supposed to run after the prow/images* jobs completed.. Might there be some "timing" issue?

[openshift-ci]

@NickLucche: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/smoke-test	86e5339	link	true	/test smoke-test
ci/prow/images	86e5339	link	true	/test images

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[dtrifiro]

Given that the image mirror steps succeed and the smoke test fails, I suspect that the failing jobs are the ones scheduled on build05

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dtrifiro, NickLucche

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [NickLucche,dtrifiro]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment