How to use multi-GPUs with dali backend? about dali_backend HOT 2 CLOSED

Hello @lbhli !

Your question is related also to the question, how do you use Triton in the multi-GPU setup? By default, if you have multi-GPU system, Triton runs one model instance per the GPU. Please refer to the instance groups tutorial to know more about how Triton works with multi-GPU system.

Going back on topic, by default Triton will run one DALI model per GPU in your system. If you like to control this behaviour, you can do it fully with instance_group setting in DALI Model Configuration. In case of Triton, it doesn't matter what you pass as device_id to the DALI pipeline definition, since Triton is going to manage it anyhow.

from dali_backend.

Hello @lbhli !

Your question is related also to the question, how do you use Triton in the multi-GPU setup? By default, if you have multi-GPU system, Triton runs one model instance per the GPU. Please refer to the instance groups tutorial to know more about how Triton works with multi-GPU system.

Going back on topic, by default Triton will run one DALI model per GPU in your system. If you like to control this behaviour, you can do it fully with instance_group setting in DALI Model Configuration. In case of Triton, it doesn't matter what you pass as device_id to the DALI pipeline definition, since Triton is going to manage it anyhow.

That's cool, make sense, thanks.
Close this issue.

from dali_backend.