Legion: collective instance freeze on slingshot-11 about legion HOT 7 OPEN

Sorry I meant 4 ranks/node. Not 1 rank/node.

from legion.

1. What scale is required to hit this freeze? (Nodes, ranks, GPUs/rank?)
2. Can you confirm that the collective instance freeze definitely does not occur on at least one other network (e.g., Infiniband)? Ideally at a similar scale to above. This would help rule out a potential Legion issue.

from legion.

The subranks branch starts to freeze at 8 nodes. The tdb branch starts to freeze at 4 nodes, 4 ranks/node, 2 gpus/rank. I can try tdb on sapling. It was working on blaze the last time I tried, but blaze is currently down.

from legion.

Actually looks like blaze is back up. Will try it there.

from legion.

The network drive on blaze is still down so I couldnt run there, but I built and ran tdb on sapling. Ran on all 4 nodes, 4 ranks/node, 1 gpu/rank. It started up and ran fine. This is probably a slingshot issue.

from legion.

Sapling has 4 GPUs per node. Could we run 16 ranks, 4 ranks/node, 1 GPU/rank?

from legion.

I'm going to assume this is a Slingshot issue unless we can reproduce it on an Infiniband machine.

from legion.