Problems with "no cuda-capable device is detected" after Ubuntu upgrade about libnvidia-container HOT 10 CLOSED

Can you try setting user = "root:vglusers" in /etc/nvidia-container-runtime/config.toml

from libnvidia-container.

Some of the presumably irrelevant hits pointed to wrong device permissions, which I tried to rule out.

crw-rw---- 1 root vglusers 195,   0 Okt  9 16:27 /dev/nvidia0
crw-rw---- 1 root vglusers 195, 255 Okt  9 16:27 /dev/nvidiactl
crw-rw---- 1 root vglusers 195, 254 Okt  9 16:27 /dev/nvidia-modeset
crw-rw-rw- 1 root root     236,   0 Okt  9 16:27 /dev/nvidia-uvm
crw-rw-rw- 1 root root     236,   1 Okt  9 16:31 /dev/nvidia-uvm-tools

I am in the vglusers group, and I also temporarily changed them all to mode 0666.

from libnvidia-container.

I found the culprit: At some point in the past, I had virtualgl installed. Uninstalling that left two configuration files, which I had to remove:

/etc/udev/rules.d/99-virtualgl-dri.rules contained

KERNEL=="card[0-9]", MODE="0660", OWNER="root", GROUP="vglusers"

and /etc/modprobe.d/virtualgl.conf contained:

options nvidia NVreg_DeviceFileUID=0 NVreg_DeviceFileGID=1005 NVreg_DeviceFileMode=0660

I removed both files, ran update-initramfs -u in order to let the changed options take effect, and did delgroup vglusers (which was 1005, of course), and now I could revert the definition of user in /etc/nvidia-container-runtime/config.toml.

from libnvidia-container.

So nvidia-container-cli info works but docker run doesn't?
Can you always reproduce it? That's weird, both of them should either work or fail.

from libnvidia-container.

Yes, exactly. docker run with nvidia runtime always fails, but I find no other problem.

from libnvidia-container.

Jackpot! 👍 At last, I am able to run nvidia-docker again! :-)

I must admit that I am still very puzzled – after all, I even tried with rw permissions for "other". Furthermore, I know several people with Ubuntu 18.04 who did not seem to have to do this (but I will double-check again, now that I know more).

from libnvidia-container.

Ok, just confirmed: on a machine with 18.04 installed from scratch and NVRM version 390.77, the config entry is missing (actually, default is a commented out #user = "root:video"), and dev permissions are:

crw-rw-rw- 1 root root 195,   0 Sep 24 13:59 /dev/nvidia0
crw-rw-rw- 1 root root 195,   1 Sep 24 13:59 /dev/nvidia1
crw-rw-rw- 1 root root 195,   2 Sep 24 13:59 /dev/nvidia2
crw-rw-rw- 1 root root 195,   3 Sep 24 13:59 /dev/nvidia3
crw-rw-rw- 1 root root 195, 255 Sep 24 13:59 /dev/nvidiactl
crw-rw-rw- 1 root root 195, 254 Sep 24 17:43 /dev/nvidia-modeset
crw-rw-rw- 1 root root 238,   0 Sep 24 13:59 /dev/nvidia-uvm
crw-rw-rw- 1 root root 238,   1 Sep 24 17:43 /dev/nvidia-uvm-tools

from libnvidia-container.

Yeah, permissions of the device files are dictated by the kernel module parameters and those can vary depending on how you install the driver. Hence the setting ;)

from libnvidia-container.

Thanks again. For completeness, I asked another colleague who has the same upgrade history AND the same driver version, and STILL the permissions are different, and she does not need that setting.

from libnvidia-container.

Many thanks @3XX0 for the support, #37 (comment) solved the problem for us. Is there anything else to consider if we want to use nvidia-docker and VirtualGL on the same machine?

from libnvidia-container.