Training: Learning Rate schedule based on iterations rather than epochs about graphcast HOT 4 CLOSED

Thanks for your message, so actually it is not just that we adjusted the learning rate as a function of iteration, rather than epoch, but actually also did not even train with proper epochs. We simply, at each iteration, sampled 32 examples from the longer ground truth trajectory, randomly and without replacement. So we only have "epochs" on average, but nothing prevented the same sequence to be sampled twice before some other sequence was sample once.

We believed this did not matter in our case because an epoch is about 54k training examples, and we trained with 300k steps batch 32 each, which corresponds to about ~180 epochs, so it probably does not matter if some examples are sampled a bit more than 180 times, and some other a bit less than 180 times.

In the case of a small dataset probably it still does not matter much so long as the number of training iterations is still large compared to the dataset size, and the learning rate decay is very slow (like in our case), but of course if you are getting to limit where each example is only seen a few times, and the learning rate decays quickly, I think it makes sense to do it in the way you are proposing.

from graphcast.

Thank you for providing clarification and detailing the training process. Understanding the methodology behind your model training is insightful. Your point about smaller datasets and quicker learning rate decay suggests that an approach based on epochs might be more suitable in those scenarios.

It seems that the issue has been satisfactorily resolved.

from graphcast.

Hello, I have some questions about model training. Have you tried training models with different resolutions, GraphCast_small (13levels, 1°) and GraphCast (37levels, 0.25°)? How much time and memory does it take to train these two models?

I look forward to your response. Thank you.

Best regards！

from graphcast.

@zhongmengyi I have replied in your separate issue #77

from graphcast.