Comments (5)
anderskm
commented on September 7, 2024
1
Hmm... that's odd.
A quick test on my own computer shows similar results.
At first, I thought it was because, GPUtil gets status of the GPUs through spawning a subprocess of nvidia-smi, while pynvml uses the C API provided by Nvidia. But that does not seem to be the case.
It would seem, that the cause is numpy, as simply importing it causes the CPU usage to rise to around 400%. Creating a script, which imports numpy and nothing else, increases the CPU usage to above 500%.
In short; numpy should probably not be used in this package. The integration of it in the package is not huge, but it's still enough that I won't have time to look at it for the next couple of weeks.
I'll keep this issue open until a solution without numpy is pushed.
With current code:
Command being timed: "python2 GPUtil-test.py"
User time (seconds): 0.10
System time (seconds): 0.19
Percent of CPU this job got: 423%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.06
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 21000
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 6450
Voluntary context switches: 26
Involuntary context switches: 29
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
Remove everything not needed by getGPUs():
Command being timed: "python2 GPUtil-test.py"
User time (seconds): 0.14
System time (seconds): 0.16
Percent of CPU this job got: 434%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.07
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 21064
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 6470
Voluntary context switches: 31
Involuntary context switches: 22279
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
Numpy calls removed, but still imported:
Command being timed: "python2 GPUtil-test.py"
User time (seconds): 0.14
System time (seconds): 0.16
Percent of CPU this job got: 434%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.07
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 21064
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 6470
Voluntary context switches: 31
Involuntary context switches: 22279
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
Numpy calls and import removed:
Command being timed: "python2 GPUtil-test.py"
User time (seconds): 0.00
System time (seconds): 0.00
Percent of CPU this job got: 60%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.02
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 6840
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 2455
Voluntary context switches: 6
Involuntary context switches: 4
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
Only importing numpy and doing nothing else:
Command being timed: "python2 numpy-test.py"
User time (seconds): 0.14
System time (seconds): 0.19
Percent of CPU this job got: 515%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.06
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 20472
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 3092
Voluntary context switches: 21
Involuntary context switches: 31
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
numpy-test.py
import numpy
from gputil.
anderskm
commented on September 7, 2024
1
Forgot to include my test of calling just nvidia-smi.
nvidia-smi.py
from subprocess import Popen, PIPE
p = Popen(["nvidia-smi","--query-gpu=index,uuid,utilization.gpu,memory.total,memory.used,memory.free,driver_version,name,gpu_serial,display_active,display_mode,temperature.gpu", "--format=csv,noheader,nounits"], stdout=PIPE)
stdout, stderror = p.communicate()
output = stdout.decode('UTF-8') Command being timed: "python2 nvidia-smi-test.py"
User time (seconds): 0.02
System time (seconds): 0.00
Percent of CPU this job got: 80%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.02
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 6756
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 2436
Voluntary context switches: 6
Involuntary context switches: 2
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
Calling nvidia-smi with the same arguments as GPUtil directly from the terminal:
Command being timed: "nvidia-smi --query-gpu=index,uuid,utilization.gpu,memory.total,memory.used,memory.free,driver_version,name,gpu_serial,display_active,display_mode,temperature.gpu --format=csv,noheader,nounits"
User time (seconds): 0.00
System time (seconds): 0.00
Percent of CPU this job got: 50%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.01
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 5744
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 791
Voluntary context switches: 4
Involuntary context switches: 1
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
from gputil.
anderskm
commented on September 7, 2024
1
It turned out to be easier, than I had anticipated :-)
I have a commit ready, but I still need to test it on a machine with multiple GPUs, before pushing it to github.
I've included some results below.
Old version using Numpy
Command being timed: "python2 GPUtil-test.py"
User time (seconds): 0.16
System time (seconds): 0.51
Percent of CPU this job got: 303%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.22
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 21176
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 45
Minor (reclaiming a frame) page faults: 6693
Voluntary context switches: 247
Involuntary context switches: 97251
Swaps: 0
File system inputs: 18744
File system outputs: 24
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
New version (using math and random)
Command being timed: "python2 GPUtil-test.py"
User time (seconds): 0.03
System time (seconds): 0.01
Percent of CPU this job got: 82%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.05
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 9364
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 2771
Voluntary context switches: 6
Involuntary context switches: 4
Swaps: 0
File system inputs: 0
File system outputs: 24
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
from gputil.
anderskm
commented on September 7, 2024
1
@konomikitten I've pushed a commit (e81a38c), which does not import numpy.
from gputil.
konomikitten
commented on September 7, 2024
1
@anderskm awesome, thank you. 😃
from gputil.
Related Issues (20)
- Crashing if nvidia-smi fails HOT 3
- GPUtil.showUtilization does not work for individual attrList HOT 3
- ImportError: No module named GPUtil HOT 4
- getFirstAvailable(maxMemory=0.9) inconsistent with showUtilization() HOT 1
- Get GPUs that are not used by any other user
- GPU memoryUsage per Process
- Is it possible to get the CUDA version? HOT 1
- Add Kubernetes support through device plugins
- Unable to find GPU on Windows HOT 11
- Request: Add all query information from nvidia-smi
- ValueError when nvidia-smi finds no GPU
- Over 60 times slower than nvidia-smi to asses resource usage
- GPUtil doesn't find GPU HOT 1
- showUtilization causes GPU stuttering
- ValueError: invalid literal for int() with base 10: 'No devices were found' HOT 1
- Drop dependency on distutils to support python 3.12 HOT 1
- Pyinstaller exe with console=False causes pop-up window every time nvidia-smi.exe is called HOT 1
- Very new to all of this, please help? HOT 1
- No longer works on Python 3.12 as distutils has been deprecated and removed HOT 2
- Handle nvidia-smi non-zero exit status
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gputil.