Coord check looks good, but μTransfer is not working as expected about mup HOT 6 CLOSED

Hi Edward, thank you very much, your advice saved me. A larger learning rate exposed the problem, the plot showed jitter, I debugged and fixed the problem, and now the plot is smooth and shows good increase in value. I'm going to try the big learning rate μTransfer on this.

Although it looks good now, the working principle is too hard for me, and the muP is really amazing.

from mup.

Thanks for your patience, Jianbin.

There are many considerations when training a very large model. In some sense, mup is a necessary but insufficient condition for successful training of large models. Other factors include the use of weight decay and floating point precision. Hope this can help with your investigation!

Another question is that the transformer example and mutransformers use different initialization methods, (init_std / d_model) ** 0.5 vs init_std * width_mult ** -0.5, are these two formulas equivalent in some sense? Will there be pros and cons?

They are equivalent up to a constant.

from mup.

Hi shjwudp,

Thanks for your interest in our work!

Your coordinate check plots seem identical across time steps, which is a sign that the learning rate is too small for the function to change. Can you try rerunning with a larger learning rate? It's possible that with a moderately larger learning rate, the muP run might blow up after a couple steps, in which case we can look into it further.

from mup.

Hi, @edwardjhu I've recently done some experiments, an extension of the previous discussion. I found that transferring the same hyperparameters from a 350M model to 1.3B scale works fine, but transferring to a larger model 2.7B blowup, does that mean my parameters are too aggressive? how should i avoid this?

My coord:

The same hyperparameters, 1.3B model and 2.7B model comparison: https://tensorboard.dev/experiment/RirdggEZS8O2rRU9clEy0g/#scalars

from mup.

Another question is that the transformer example and mutransformers use different initialization methods, (init_std / d_model) ** 0.5 vs init_std * width_mult ** -0.5, are these two formulas equivalent in some sense? Will there be pros and cons?

from mup.

Hi Edward, thank you very much, your advice saved me. A larger learning rate exposed the problem, the plot showed jitter, I debugged and fixed the problem, and now the plot is smooth and shows good increase in value. I'm going to try the big learning rate μTransfer on this.

Although it looks good now, the working principle is too hard for me, and the muP is really amazing.

Hi Jiabin, Could you share info on what caused the jitter in your coord check plots? Its possible that I have a similar issue (#58).

Thanks!

from mup.