Can't use GPU in the local mode. about pocketflow HOT 12 CLOSED

Thanks for the detailed description.
This is caused by the same problem as discussed in issue #29. PocketFlow failed to use the GPU device due to the over-strict definition of "idle GPU" in utils/get_idle_gpus.py. In that Python script, only GPUs without any running processes are defined as idle GPUs. This is often not the case, if the GPU is also used for desktop rendering.

We will fix this problem ASAP. Sorry for your trouble.

from pocketflow.

Enhancement required: change the way of detecting available GPUs.

from pocketflow.

@jiaxiang-wu Thanks for reply and explanation. And I realized that why the test passed on a machine with 4 GPUs and failed on this machine with one GPU. I will follow your update and test later.

from pocketflow.

@jiaxiang-wu In addition, I hope you could notice me after fixing the problem. Thanks.

from pocketflow.

@howtocodewang No problem. We will update this issue after the fix is done, and also send you an e-mail as notification.

from pocketflow.

Will be glad to send a PR fixing this. I would like to have a clarification on the new policy though.
What should be termed as an idle GPU? [Currently..a GPU with no processes running is defined as idle gpu in utils/get_idle_gpus.py].

from pocketflow.

@KranthiGV In my opinion, an idle GPU which is defined by authors is a GPU with no any processes even as a display device. That means if you only have one GPU in your PC, your GPU will process some display tasks at the same time while you are running some GPU-based algorithm. In this situation, your GPU is not an idle GPU. So I encountered this issue on a PC with one GPU device but ran the demo successfully on a PC with 4 GPUs.

from pocketflow.

@howtocodewang Yes, you are right.
@KranthiGV How about a GPU with at least 50% GPU memory free? Or, if multiple GPUs are presented, return the GPU with the most free memory? What's your opinion?

from pocketflow.

1. At least 50% free memory of GPU seems to be a good assumption since display processes won't take more than 50%. They usually take only a few hundreds of MB.
2. It multiple GPUs are present, returning GPU with most free memory is not desirable. Currently, we return a list of GPUs (example 0,1,2,3 or 0,1, etc). Returning only the GPU with most free memory would prevent multi-GPU usage.

I'll go ahead with implementing 1st policy.

from pocketflow.

@KranthiGV Great, looking forward to your contribution.

from pocketflow.

@jiaxiang-wu
I cannot find dev branch to send the PR. There's only master.
Should I raise PR for master branch?
My fix is here: idle_gpu_fix

from pocketflow.

@KranthiGV
Yes, please raise a pull request to the master branch.

from pocketflow.