deepvariant 1.6.0 with singularity gpu support about deepvariant HOT 5 CLOSED

Hi @melop , I'll be helping you with this, but I'm a bit behind on my tasks. I'll assign this myself and I'll try to take a look soon (hopefully by end of tomorrow!)

from deepvariant.

Hi @melop , sorry that it took me a while.

I have a question for you: When call_variants step is running, is your GPU being utilized?

from deepvariant.

And @melop , the most recent time I tried to set up a GPU machine was using these steps:

#745 (comment)

I don't think I've seen the failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected error message you saw.

Based on that error message I did a bit internet search. I wonder if this is relevant: https://stackoverflow.com/a/48715413 Specifically , try setting CUDA_VISIBLE_DEVICES to 0, by running export CUDA_VISIBLE_DEVICES=0 and see if that works for you?

from deepvariant.

Hi @melop , let me know if you have a chance to try and follow up. I'll close this because it has been 2 weeks. Please feel free to follow up and reopen if you have more questions.

from deepvariant.

I get the same error. There appears to be Cuda 12 modules being used even though Cuda 11 is installed? Adding '--env LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/lib/python3.8/dist-packages/nvidia/cublas/lib:/usr/local/lib/python3.8/dist-packages/nvidia/cuda_nvrtc/lib:/usr/local/lib/python3.8/dist-packages/nvidia/cuda_runtime/lib:/usr/local/lib/python3.8/dist-packages/nvidia/cudnn/lib' lets the image see all these libraries and stops the "Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libcublas.so.12: cannot open shared object file" error.

But now during the call_variants step, I get a new error --> "2024-03-11 23:23:22.756430: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:433] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
2024-03-11 23:23:22.756490: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:442] Possibly insufficient driver version: 470.57.2"

Just doing '--env LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/lib/python3.8/dist-packages/nvidia/cublas/lib' also seems to get past the "Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libcublas.so.12:" error, but now I get "could not retrieve CUDA device count: CUDA_ERROR_NOT_INITIALIZED: initialization error" during the call_variants step

The 1.6.0-gpu image appears to have some mixed cuda 11 & cuda 12 module library conflict issues. Some of these issues may be suppressed if you have a sufficient nvidia driver for both cuda 11 and cuda 12 (I'm still waiting to test that though).
Note that 1.5.0 has no issues running for me.

from deepvariant.