Can't use one GPU? about moco HOT 8 CLOSED

We exploit the fact that by default BN is split into multiple GPUs in which mean/std is independently computed. To run on one GPU, you may implement BN split along the N (batch) dimension to mimic this effect (see NaiveBatchNorm in Detectron2). You also need to change the lr (e.g., linearly) if you change the batchsize to fit memory. To run on 2 GPUs, try --lr 0.0075 --batch-size 64.

from moco.

implement BN split along the N (batch) dimension to mimic this effect

There is an implementation of split (aka sub-batch) BN for video (3d) models that might be helpful to you:
https://github.com/facebookresearch/SlowFast/blob/master/slowfast/models/batchnorm_helper.py#L37-L46

This can easily be adapted to the the 2d BN case.

from moco.

@rbgirshick Thanks for the link. I actually needed this and am writing a code myself starting from there. BTW, if you know, It would be great if you could answer this question, which is regarding the code snippet. facebookresearch/SlowFast#357 (comment)

from moco.

@rbgirshick Thanks for the link. I actually needed this and am writing a code myself starting from there. BTW, if you know, It would be great if you could answer this question, which is regarding the code snippet. facebookresearch/SlowFast#357 (comment)

@rbgirshick Thanks for the link. I actually needed this and am writing a code myself starting from there. BTW, if you know, It would be great if you could answer this question, which is regarding the code snippet. facebookresearch/SlowFast#357 (comment)

Hello, have you realized this? I also want to implement MOCO on a single GPU.

from moco.

@JingLiJJ maybe you can see https://github.com/leftthomas/MoCo

from moco.

from moco.

@JingLiJJ maybe you can see https://github.com/leftthomas/MoCo

Hey! I would really want to run it on a single GPU, but I am not sure if I understand how this implementation works for a single GPU, could you please explain?

from moco.

from moco.