Issues loading CC3M about alpro HOT 5 CLOSED

What is the machine spec you use for training, e.g. how many GPU of which model and VRAM?

Does the training continue if you ignore the warning message and wait for a longer period of time?

Will the issue persist if you try to 1) change the number of workers in dataloader; 2) having more or fewer GPUs; 3) having larger or smaller batch size? 4) activating the training with mpirun (see here) ?

The discussion here could be related.

Also checkout the discussion on Hovorod repo here and here.

from alpro.

I used 8 A100 GPUs and 1T VRAM for training. I think the allgather operation might cause the deadlock? Is there any way that might solve this?

1,0]<stderr>:[2022-04-20 02:35:43.587069: W horovod/common/stall_inspector.cc:105] One or more tensors were submitted to be reduced, gathered or broadcasted by subset of ranks and are waiting for remainder of ranks for more than 60 seconds. This may indicate that different ranks are trying to submit different tensors or that only subset of ranks is submitting tensors, which will cause deadlock. [1,0]<stderr>:Stalled ranks: [1,0]<stderr>:0: [allgather.noname.4624] [1,0]<stderr>:1: [allgather.noname.4624] [1,0]<stderr>:4: [allgather.noname.4624] [1,0]<stderr>:6: [allgather.noname.4624] [1,0]<stderr>:7: [allgather.noname.4624]

from alpro.

Could you try the possible fixes I suggested above and also check out the related threads?

This is not necessarily a deadlock. More likely it is just because some ranks are busier while keeping others waiting.

from alpro.

Thanks, I have figured out. When loading the images, I use while True to avoid non-existing images. Besides, I have a question about the weight decay factor. Did you use the weight decay during the both pre-training and finetune stage? Since I have noticed ALBEF uses the weight decay but not ALPRO.

from alpro.

Glad you've figured it out.

I'm closing the issue for now, as the original question has been resolved.

from alpro.