All CUDA-capable devices are busy or unavailable

[DaBeIDS]

Hi,
it seems that GPU is not fully available.

I tried on the aicoe-osc-demo names space to run infer_on_pdf.py inside of the main-terminal. The model pod should actully have a GPU attached to it and would not create itself if there is no GPU, but now i got the following error message.

Let me know if you need more details. I can not extract data currently.

[DaBeIDS]

We figured out that the pod did not hat a resource limit and therefore did not check if it really had a GPU which it can use. Due to that a GPU node was assigned which is already in use.
A temporary solution is:
spec:
..
template:
..
containers:
...
resources:
limits:
nvidia.com/gpu: '1'
But that will fix the gpu to the pod. We have to check how we can manage to turn on and off the gpu pod when it is really triggered.

[eharrison24]

Mikhail is working on this task this week 11/27 - 12/2