Distributed Initialization Fails When Pretraining with Multiple GPUs about pytorch_empirical-mvm HOT 3 CLOSED

Turns out that this issue was a result of os.environ['LOCAL_SIZE'] not being set so the local_size variable used by the NodeSplitSampler was incorrectly defaulting to 1. The variable LOCAL_SIZE doesn't appear to be a standard environment variable used in PyTorch's distributed package, is it?

Setting LOCAL_SIZE to the number of GPUs on a single node resolves the issue.

In which case, given 8 gpus on a single node, are these variables in the NodeSplitSampler expected:

world_size=8
local_size=8
node_size=1
node_idx=0
rank=GPU rank (0-7)
local_rank=GPU rank (0-7)

In doing so, am I right in understanding that the expected behavior of get_index_on_node is to assign all 10 of the composite files (0-9) for each dataset to the single node. And the expected behavior ofget_index_on_rank is to split the full dataset by indices between the 8 gpus.

from pytorch_empirical-mvm.

Thanks for pointing this out!

The distributed variables will be set automatically in our training environment, so I missed that part in this release code 😅.

from pytorch_empirical-mvm.

It should work after adding LOCAL_SIZE='8'.
I also update the command in README.

Super thanks again for your clarification 😍

from pytorch_empirical-mvm.