torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1333, remote process exited or there was a network error, NCCL version 2.18.6 about deepspeedexamples HOT 3 OPEN

@Rainbowman0, to help with further investigation, can you try running the micro benchmarks here https://github.com/microsoft/DeepSpeedExamples/tree/master/benchmarks/communication

from deepspeedexamples.

@Rainbowman0, to help with further investigation, can you try running the micro benchmarks here https://github.com/microsoft/DeepSpeedExamples/tree/master/benchmarks/communication

Thank you! I will have a try! :)

from deepspeedexamples.

OHHH!! I think it is because the socket interface error. I have solved it by below steps:
First run: ifconfig to show the socket interfaces that can be used:

(I can't use 'eno1' interface, which is strange. So i use the 'br0' finally.)

open ~/.bashrc:

vim ~/.bashrc

write these commands at the bottom:

export NCCL_IB_DISABLE=1
export NCCL_SOCKET_IFNAME=br01 # for me it is 'br0' interface, you should use yours :)
export NCCL_P2P_DISABLE=1
export NCCL_DEBUG=INFO

reload ~/.bashrc

source ~/.bashrc

Both 192.168.3.100 and 192.168.3.101 follow the same steps. :)

from deepspeedexamples.