fix: Assume 0 MiB of memory is available on a disabled NVIDIA GPU #537
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ry-gpu=memory.total returns "[N/A]" Signed-off-by: Daniel Cilevitz <[email protected]>
erico-aws
added
the
response-requested
Signed-off-by: Daniel Cilevitz <[email protected]>
cilevitz
force-pushed
the
fix-disabled-gpu-memory-detection
branch
from
e067ad2 to
e864a6f
Compare
AWS-Samuel
reviewed
Feb 10, 2025
| for line in output.splitlines(): | ||
| mem_mib = int(line.replace("MiB", "")) | ||
| mem_per_gpu.append(mem_mib) | ||
| mem_per_gpu = _parse_gpu_memory(output) |
There was a problem hiding this comment.
Suggested change
| mem_per_gpu = _parse_gpu_memory(output) | |
| mem_per_gpu = _parse_gpu_memory(output, verbose) |
I think we will want to pass on the verbose option when we call this function
There was a problem hiding this comment.
Added verbose parameter in e78b9da.
Signed-off-by: Daniel Cilevitz <[email protected]>
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…
What was the problem/requirement? (What/Why)
As described in #536:
On a system with a disabled NVIDIA GPU that has no memory allocated to the GPU, the worker agent exits in its initialization phase due to a critical error from
_get_gpu_memory()trying to parse a string as an integer.What was the solution? (How)
nvidia-smi --query-gpu=memory.totalreturns a memory value that can be parsed byint()if the GPU is active, or the string "[N/A]" if the GPU is disabled. There are no other mode flags (e.g.display_active,display_mode) that can be used to determine whether a GPU has no memory allocated to it.Catching the
ValueErrorraised when trying to parse the "[N/A]" string output fromnvidia-smi --query-gpu=memory.totalinstead of letting it propagate upward allows the worker agent to finish initializing. A GPU that's present in a system but not active contributes zero memory towards that system's total GPU memory, and therefore_get_gpu_memory()is made to return 0 for a GPU if that GPU's memory size cannot be queried.What is the impact of this change?
GPUs without allocated memory will be part of the total number of GPUs available on the machine, but they contribute 0 MiB to the total amount of GPU memory available on the machine. The worker agent emits a warning in its log that a non-numeric result was received from
nvidia-smi.How was this change tested?
The change was tested on an affected machine and the worker agent starts up successfully.
Was this change documented?
The docstring is updated and the code block has comments.
Is this a breaking change?
No.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.