I can't get the results as you provided with 4 rdma machines. about byteps HOT 5 CLOSED

Do you observe high variation on the training speed? Or is it consistently low? I think you are taking the average value. Can you please show a section of the log output (containing multiple lines of the training speed) of the workers?

from byteps.

@MarvinLong

There are a few things you can try. If any of the following works for you, please let us know. Though the following env may start with MXNET, they apply to any workers, TF/MXNet/PyTorch, because the parameter server is based on MXNet.

1. For the parameter servers, set export MXNET_OMP_MAX_THREADS=10 if you have 16 CPU cores per server. Set export MXNET_OMP_MAX_THREADS=4 if you only have 8 CPU cores

2. Set export MXNET_CPU_WORKER_NTHREADS=32 . This may speed up the parameter server

3. Start more parameter server instances. For example, when you have two physical machines to run the servers, you can start 4 (DMLC_NUM_SERVER=4), two server instances per physical machine. This will increase the network bandwidth utilization especially when your single TCP flow cannot saturate your bandwidth.

from byteps.

@bobzhuyb thank you for your reply.
All my rdma machines are in use for training a bert. I will try it next week.
Actually I want to use this kit for training a bert, but when I use this to test a bert_large model (vocab size is 110k) training speed. The speed up ratio is much lower than horovod’s . I will upload my experiment results next week.

from byteps.

@MarvinLong Do you have NVLinks? If not, BytePS may not give you speed-up because it is optimized for the scenario where the network bandwidth is the bottleneck. However, with 80Gbps RDMA without NVLinks, the bottlenck is inside each machine.

If you have NVLinks, properly configured BytePS should give you significant speedup compared with Horovod, especially with 3+ workers.

from byteps.

@bobzhuyb yes, I have 4 NVLinks machines with IB. As your advise I can use them as 2 workers and 2 servers.

from byteps.