Clip norm after scaler.unscale_ in native fp16 training

[del2z]

🐛 Bug

I trained a large model using native amp, but the loss converged very slow. After a careful check of the backward and optimization code, I found the clip_gradients is executed right after backward, but scaler.unscale_ is conducted in pre_optimization_step.
According to the instruction of Pytorch, the order of clip and unscale should be exchanged. Currently gradient_clip_val may lead to a very flat learning curve if used together with native amp.
Hope to be fixed.

[justusschock]

Hey @del2z ,

thanks for letting us know. I just checked and it seems you are right. I'll get to work on this ASAP :)

[cowwoc]

I believe this is fixed in release 1.4.9: https://github.com/PyTorchLightning/pytorch-lightning/releases/tag/1.4.9

Can this issue be closed?