[Feature] Support a custom e2e validation

[changhyuni]

Issue or feature description

I sometimes elastically manage GPU nodes to save costs, creating and deleting them as needed. It would be helpful to have a validation feature that assigns a GPU to a real pod to ensure it functions correctly when a node is provisioned and validated.

Questions:

Is this possible? If enabled, after all the basic validations pass, the nvidia-operator could taint the node, deploy a pod (e.g., an application called e2e-check), and assign the GPU to perform a specific task, such as a basic inference or any workload utilizing the GPU. If the gpu operate good, the taint would be removed from the node.

Or, is there an existing method that currently supports this? I could implement it myself, but it would be good if a validation feature like this could be added to the nvidia-operator.

[Hexoplon]

The gpu-operator-validator component supports perforing cuda validation by running a workload, by setting the WITH_WORKLOAD env variable to "true". This might help with what you are looking for, the values is listed in the default helm chart values file