Best GPU device 1 detected, but training still runs on less powerful GPU 0 device? about deepfacelab HOT 14 CLOSED

@iperov Need to re-open this issue. I have a GTX 980 Ti in slot 0 and a GTX 1080 in slot 1 but training/extracting uses the wrong GPU and allocates the wrong amount of VRAM.

When I use --force-worst-gpu it does the opposite, displays the 980 Ti and allocates 6GB but runs on the GTX 1080 (as seen below)

from deepfacelab.

actually there is no --force-worst-gpu param

from deepfacelab.

Sorry it is actually --choose-worst-gpu but that doesn't really affect it being a problem.

from deepfacelab.

Cuda 10 ? TF 1.11.0 doesnt support cuda 10.
Looks like you manually built prealpha version of tf 1.13.0 which is possible buggy to use with keras multigpu model.

"Created_vram_gb" used just indicator for models to use normal version of models or low-mem version. It doesnt operate gpus.

from deepfacelab.

It is tensorflow 1.12

from deepfacelab.

same tf 1.12 doesnt support cuda 10.

from deepfacelab.

try using tf 1.11 and cuda 9 as current repo requirements

from deepfacelab.

Downgraded to Tensorflow 11.1 and CUDA 9, problem persists.

from deepfacelab.

i dont know how to help you

from deepfacelab.

TheFeelTrain, I dealt with the issue by putting the 1080ti in PCIe slot 0, and the 1070 in PCIe slot 1. As currently written, the code seems to default to the GPU in PCIe slot 0, regardless of whether it's the "best" GPU or not, at least on my Ryzen motherboard. As iperov says, try creating a new environment with python=3.6, CUDA 9, Cudnn 7.05 and TF 1.11, the official requirements of the repro.

from deepfacelab.

I can't swap the slots because I want to use the 980 Ti as my main GPU while I train on the 1080. The code needs to be changed to choose the actual best card instead of defaulting to slot 0.

Also the Python, CUDA, and Tensorflow versions do not affect this behavior. I downgraded everything (breaking a lot of other programs and drivers in the process) only for it to be exactly the same as it was before. The actual program runs fine for me with CUDA 10 and Tensorflow 1.12 anyways. Maybe the AUR version of Tensorflow was compiled to use CUDA 10.

from deepfacelab.

The code needs to be changed to choose the actual best card instead of defaulting to slot 0.

then change it, if you know what to change.

from deepfacelab.

I don't know python. You're supposed to be the developer here :/

from deepfacelab.

TheFeelTrain, the workaround is to put your 1080ti in PCIe slot 0, if you want to use it for training. If you're using Windows 10 and are concerned about VRAM reservation, connect your monitor(s) to the 980 Ti and install the repro under Ubuntu, which has better management of GPU resources.

from deepfacelab.