Enhance Nvidia Device plugin with more health checking features about k8s-device-plugin HOT 4 OPEN

what GPU Allocated memory checks and "Zombie processes" checks do?

Those are related, and it's related to reset too. If it's not possible to reset your card, you at least need to detect when things are broken.

When things go awfully wrong, you can have the following:

• There is no process using the GPU, but nvidia-smi shows a non-trivial amount of memory being used. e.g. something like 352MiB / 12181MiB
• There is a process already using the GPU you are supposed to give to a new container (excluding voluntary sharing). This can happen when there is a GPU fault and the process that had an open CUDA context can't teardown properly.

These kinds of checks are useful safety checks in addition to event-based healthchecks like XIDs and ECCs. Some of these errors could go unnoticed otherwise.

from k8s-device-plugin.

@flx42 @jiayingz any plan to enhance current device health check?

from k8s-device-plugin.

cc @jiayingz @vishh @mindprince

from k8s-device-plugin.

@flx42 Thanks for your detailed explanation, it's very useful for us!

from k8s-device-plugin.