Gradient clipping

Capping the norm of the gradient before the optimiser step, so one freak batch cannot take a huge step and destroy the run. Standard practice in language model training.

A definition is not an understanding. This is taught properly in Derivatives and the chain rule, part of Calculus for ML.

Learn this properly

Terms from the same topic

Gradient descentStochastic gradient descentAdamGradientPartial derivativeChain rule