Questions for training gpt-2 using mup about mup HOT 6 CLOSED

Can you share the coordinate check plots like these?

https://github.com/microsoft/mup/tree/main/examples/Transformer/coord_checks

They are really helpful for debugging.

from mup.

The coordinate check plots are given below. It seems the Mup's plot is much more stable than SP's. However, in the case of high learning rate, sometimes there will be some fluctuations in Mup's plot, and I am not sure whether these fluctuations have an impact on Mup.

• SP, Adamw, lr=0.001
  

• SP, Adamw, lr=0.01
  

• SP, Adamw, lr=0.1
  

• Mup, Adamw, lr=0.001
  

• Mup, Adamw, lr=0.01
  

• Mup, Adamw, lr=0.1
  

from mup.

Can you extend the x-axis to the right as much as you can for mup?

from mup.

Can you extend the x-axis to the right as much as you can for mup?

Now, I have tested the width from 48 to 3072 with the base shape = 768.

• SP, adamw, lr=0.001
  

• Mup, adamw, lr=0.001
  

from mup.

I've identified the problem. Thanks~

from mup.

@jiangjiadi Could you share what the problem was and how you remedied it?

from mup.