Implementation error about sam HOT 5 CLOSED

Hi,

you are right that taking the norm of concatenated parameters would work but it is a very costly operation for big models. This is why I divide the computation and calculate the L2 norm separately for each layer. I believe that this optimization is correct, do you have any counterexample?

I don't want to make a rigorous proof but here is an example to show you the intuition behind that code: notice that

On top of that, even PyTorch uses the same trick for gradient clipping :)

Cheers,
David

from sam.

Hi David,
my bad, thanks for clarification. I did not notice that those are equal.
Have a nice day :)
Cheers,
Jacek

from sam.

Hi, I am confused that should we calculate grad norm for each parameter separately or calculate it for all parameters?

from sam.

Hi, I am confused that should we calculate grad norm for each parameter separately or calculate it for all parameters?

Same question. Do you have any insight?

from sam.

We calculate the norm of the gradients from all parameters, as explained above.

from sam.