module 'pynvml' has no attribute '_nvmlGetFunctionPointer' #153

BEpresent · 2023-05-04T12:40:18Z

Describe the bug

Calling gpustat leads to the following error:

Screenshots or Program Output

Traceback (most recent call last):
  File "/home/be/.local/bin/gpustat", line 5, in <module>
    from gpustat import main
  File "/home/be/.local/lib/python3.9/site-packages/gpustat/__init__.py", line 16, in <module>
    from .core import GPUStat, GPUStatCollection
  File "/home/be/.local/lib/python3.9/site-packages/gpustat/core.py", line 24, in <module>
    from gpustat.nvml import pynvml as N
  File "/home/be/.local/lib/python3.9/site-packages/gpustat/nvml.py", line 57, in <module>
    _original_nvmlGetFunctionPointer = pynvml._nvmlGetFunctionPointer
AttributeError: module 'pynvml' has no attribute '_nvmlGetFunctionPointer'

Please provide the output of gpustat --debug and nvidia-smi. Or attach screenshots if applicable.

Environment information:

OS: Debian 11.7
NVIDIA Driver version: NVIDIA-SMI 470.182.03 Driver Version: 470.182.03 CUDA Version: 11.4
The name(s) of GPU card: 2 T4
gpustat version: gpustat --version doesn't work, but I did pip install it so I assume it fetches the latest version Successfully uninstalled gpustat-1.1
pynvml version: nvidia-ml-py 11.525.112

Additional context

It should be noted that it has worked 2-3 weeks ago, but now it does not on 3 different machines I have tested.

The text was updated successfully, but these errors were encountered:

wookayin · 2023-05-07T18:33:29Z

Thanks for the report. Hmm, I see _nvmlGetFunctionPointer still exists in 11.525.112 version:

https://github.com/wookayin/nvidia-ml-py/blob/11.525.112/pynvml.py#L853

Could you please try opening the file /home/be/.local/lib/python3.9/site-packages/pynvml.py and see if the function exists? It should be at around Line 852.

herrenP · 2023-05-10T08:11:40Z

This fixed my issue at least:

~ > pip install --force-reinstall gpustat
Defaulting to user installation because normal site-packages is not writeable
Collecting gpustat
  Downloading gpustat-1.1.tar.gz (97 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.9/97.9 kB 2.0 MB/s eta 0:00:00
...removed lines for brevity...
Successfully installed blessed-1.20.0 gpustat-1.1 nvidia-ml-py-11.525.112 psutil-5.9.5 six-1.16.0 wcwidth-0.2.6

~ > gpustat                                                                                     
[0] NVIDIA GeForce RTX 3070 | 52°C,  21 % |   553 /  8192 MB | ...

wookayin · 2023-05-10T17:59:34Z

TL;DR) force-reinstall nvidia-ml-py; pip install --force-reinstall --ignore-installed nvidia-ml-py

Thank you for the information. Yes, the pynvml should be the one that you get from your site-packages:

$ python -c 'import pynvml; print(pynvml.__file__)'
/usr/lib/python3.10/site-packages/pynvml.py

So apparently this pynvml seems to be coming from an old version of pynvml. This can happen, for instance, some other python packages install legacy nvidia-ml-py3 which might override pynvml; pip list can't tell this. So that's why I'm asking the checksum of the pynvml file to ensure the package hasn't been corrupted.

Although this is an user problem (or one of the downstream packages used in your project), I will try to add some fool-proof error messages when pynvml isn't alright.

wookayin · 2023-05-17T07:56:54Z

According to ray-project/ray#34196 (/cc @yuanwu2017), the problematic pynvml file comes from this third-party fork(?):

https://github.com/gpuopenanalytics/pynvml

~~Actually this is published nowhere so I have no idea why users end up having this conflicting package.~~ https://pypi.org/project/pynvml/ this should not be used.

mattip · 2023-05-17T08:41:59Z

Hmm. Is there any way to distinguish between nvidia-ml-py and pynvml, and perhaps raise a helpful error or somehow integrate the two APIs? Looking at the download stats, pynvml has about 113,500 / week, nvidia-ml-py has about 390,500 / week, so there are a lot of people using pynvml. I can understand them, anyone looking for the "natural" project to supply pynvml would assume the project name is pynvml not nvidia-ml-py.

wookayin · 2023-05-17T09:12:17Z

nvidia-ml-py is the only official, right package to use. But it's also likely that some third-party downstream packages use the wrong one. It's possible to detect which one is used so I will try to add some warning messages. Note that this error is happening due to the monkey-patching used in #143 which are trying to monkey-patch a private API of pynvml; the gpuopenanalytic's pynvml package mostly works the similar is not exposing private APIs. (I think this package should not have existed if they are going to provide the same API...)

mattip · 2023-05-17T09:47:53Z

here are the packages github detects as depending on pynvml. There are 156, some with over 1000 stars. I opened issues on the ones with lots of stars. 😄

wookayin · 2023-05-17T10:19:58Z

I've added some error messages. It would be not difficult to make gpustat compatible with the non-official pynvml package, but let me figure out a good solution when I have more spare time.

An un-official distribution of the pynvml package provided by @gpuopenanalytics/pynvml was causing an incompatibility issue, where gpustat.nvml fails to import. Although the official and correct pynvml packages should be installed, but wild users' python environments can be easily messed up by unawaringly installing the problematic, unofficial pynvml package. gpustat can support this third-party pynvml by directly importing the low-level pynvml.nvml package (which should be exactly the same as an official pynvml.py). This is a quickfix that relies on the implementation detail of a third-party package, but nevertheless it should be a good enough workaround solution for many users in practice.

wookayin · 2023-06-01T01:40:23Z

7c09a0f makes gpustat compatible even with @gpuopenanalytics/pynvml. Closing this as fixed.

wookayin · 2023-06-21T01:16:55Z

You'll also need to do pip uninstall pynvml as well.

(cherry picked from commit e6bf58f, backport to 1.1.x)

An un-official distribution of the pynvml package provided by @gpuopenanalytics/pynvml was causing an incompatibility issue, where gpustat.nvml fails to import. Although the official and correct pynvml packages should be installed, but wild users' python environments can be easily messed up by unawaringly installing the problematic, unofficial pynvml package. gpustat can support this third-party pynvml by directly importing the low-level pynvml.nvml package (which should be exactly the same as an official pynvml.py). This is a quickfix that relies on the implementation detail of a third-party package, but nevertheless it should be a good enough workaround solution for many users in practice. (cherry picked from commit 7c09a0f, backported to 1.1.x)

wookayin · 2023-08-22T19:41:48Z

(backported to v1.1.1 and released)

Revert "Make gpustat.nvml compatible with a third-party fork of pynvml" This reverts commit 7c09a0f. gpustat v1.1.1 allowed the problematic 'pynvml' package to be used as a workaround, but this still causes many problems (e.g., #168). Only the official nvidia-ml-py can be used with gpustat. See #153, #168

BEpresent added the bug label May 4, 2023

This comment was marked as duplicate.

Sign in to view

This comment was marked as off-topic.

Sign in to view

wookayin mentioned this issue May 16, 2023

[dashboard] dashboard error because of gpustat version 1.1 ray-project/ray#34196

Closed

This was referenced May 17, 2023

Use the nvidia-supplied nvidia-ml-py instead of pynvml mosaicml/llm-foundry#154

Closed

Use the nvidia-supplied nvidia-ml-py instead of pynvml PaddlePaddle/PARL#1101

Open

Use the nvidia-supplied nvidia-ml-py instead of pynvml FedML-AI/FedML#1064

Closed

mattip mentioned this issue May 17, 2023

Use the nvidia-supplied nvidia-ml-py instead of pynvml NVIDIA-Merlin/core#321

Open

wookayin added a commit that referenced this issue May 17, 2023

Show error messages when incorrect pynvml is installed (#153)

ab11ceb

wookayin added a commit that referenced this issue May 17, 2023

Show error messages when incorrect pynvml is installed (#153)

e6bf58f

wookayin pinned this issue May 17, 2023

wookayin added the pynvml label May 17, 2023

wookayin closed this as completed Jun 1, 2023

This comment was marked as duplicate.

Sign in to view

wookayin mentioned this issue Jul 14, 2023

AttributeError: module 'pynvml' has no attribute '_nvmlGetFunctionPointer' gpuopenanalytics/pynvml#47

Open

This comment was marked as duplicate.

Sign in to view

cvm-a mentioned this issue Jul 24, 2023

Use the nvidia-supplied nvidia-ml-py instead of pynvml DeMoriarty/fast_pytorch_kmeans#20

Open

wookayin added a commit that referenced this issue Aug 22, 2023

Show error messages when incorrect pynvml is installed (#153)

1a627cb

(cherry picked from commit e6bf58f, backport to 1.1.x)

wookayin added this to the 1.2 milestone Aug 22, 2023

wookayin mentioned this issue Nov 9, 2023

[Core] Ray auto detect nvidia Gpu with pynvml ray-project/ray#41020

Merged

8 tasks

wookayin mentioned this issue Jan 11, 2024

Misreported used memory with the driver 535.129.03 #168

Closed

jaywonchung mentioned this issue May 8, 2024

Detect and reject unofficial pynvml bindings ml-energy/zeus#71

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

module 'pynvml' has no attribute '_nvmlGetFunctionPointer' #153

module 'pynvml' has no attribute '_nvmlGetFunctionPointer' #153

BEpresent commented May 4, 2023

wookayin commented May 7, 2023

This comment was marked as duplicate.

herrenP commented May 10, 2023 •

edited

Loading

wookayin commented May 10, 2023 •

edited

Loading

This comment was marked as off-topic.

wookayin commented May 17, 2023 •

edited

Loading

mattip commented May 17, 2023

wookayin commented May 17, 2023 •

edited

Loading

mattip commented May 17, 2023

wookayin commented May 17, 2023 •

edited

Loading

wookayin commented Jun 1, 2023 •

edited

Loading

This comment was marked as duplicate.

wookayin commented Jun 21, 2023

This comment was marked as duplicate.

This comment was marked as duplicate.

This comment was marked as duplicate.

wookayin commented Aug 22, 2023

module 'pynvml' has no attribute '_nvmlGetFunctionPointer' #153

module 'pynvml' has no attribute '_nvmlGetFunctionPointer' #153

Comments

BEpresent commented May 4, 2023

wookayin commented May 7, 2023

This comment was marked as duplicate.

herrenP commented May 10, 2023 • edited Loading

wookayin commented May 10, 2023 • edited Loading

This comment was marked as off-topic.

wookayin commented May 17, 2023 • edited Loading

mattip commented May 17, 2023

wookayin commented May 17, 2023 • edited Loading

mattip commented May 17, 2023

wookayin commented May 17, 2023 • edited Loading

wookayin commented Jun 1, 2023 • edited Loading

This comment was marked as duplicate.

wookayin commented Jun 21, 2023

This comment was marked as duplicate.

This comment was marked as duplicate.

This comment was marked as duplicate.

wookayin commented Aug 22, 2023

herrenP commented May 10, 2023 •

edited

Loading

wookayin commented May 10, 2023 •

edited

Loading

wookayin commented May 17, 2023 •

edited

Loading

wookayin commented May 17, 2023 •

edited

Loading

wookayin commented May 17, 2023 •

edited

Loading

wookayin commented Jun 1, 2023 •

edited

Loading