Error on querying NVIDIA devices | OverflowError: Python int too large to convert to C long about gpustat HOT 9 CLOSED

I encountered the same problem, and it's been solved by downgrading the nvidia-ml-py to a former version 11.525.112 using pip install nvidia-ml-py==11.525.112. I hope it's helpful.

from gpustat.

+1 same error

• OS: Windows 10 Enterprise (Version: 2004, OS build: 19041.264)
• NVIDIA Driver version: 536.99
• The name(s) of GPU card: NVIDIA GeForce RTX 4090 x 2
• gpustat version: gpustat 1.1.1

Thanks for the workaround @Lunar13737 , it worked for me.

from gpustat.

@wookayin I think it was most likely 12.535.77 that caused the error, though I'm not 100% sure because I didn't keep a record of it. I downgraded to 11.525.112 which worked, and now 12.535.108 works too.

from gpustat.

Thanks. I can conclude that the root cause of this bug is essentially same as #161: one should use neither nvidia-ml-py=11.535.77 nor broken NVIDIA drivers >= 535.43, < 535.98.

gpustat will print warnings when any of these versions of nvml library or driver is detected, so we can close this issue without adding an unnecessary compatibility layer.

from gpustat.

+1 and the workaround with downgrading nvidia-ml-py did not work for me :(

• OS: Windows 11 Pro N
• NVIDIA Driver Version: 535.98, CUDA Version: 12.2
• GPU: NVIDIA gpuGeForce RTX 4070
• gpustat version: gpustat 1.1.1

Any hints?

from gpustat.

I'd like to reproduce this issue to have a correct fix. But I've never seen the issue.

What we know from #161 (comment):

• nvidia-ml-py=11.535.77 is buggy, only works for 535.43 and 535.86 (the OP's case):
  • Does the problem go away if you install nvidia-ml-py==12.535.108? @JensWendt
• It looks like that nvidia-ml-py 12.535.108 should correct all process-information related bugs, reverting the breaking changes in the previous versions. But this is just my guess, I'm not sure. I would need the nvidia-ml-py version installed on the system.

@Lunar13737, @PyroGenesis, @mjmikulski thanks for the datapoints. Could you please try upgrading nvidia-ml-py==12.535.108 and see if the OverflowError is gone?

from gpustat.

Could you please try upgrading nvidia-ml-py==12.535.108 and see if the OverflowError is gone?

@wookayin I can confirm, overflow error does not occur in nvidia-ml-py 12.535.108

from gpustat.

@PyroGenesis Thanks. What was the previous version of nvidia-ml-py that resulted in this bug?

from gpustat.

@wookayin nvidia-ml-py 12.535.108 works for me, no overflow error

from gpustat.