Exploding gradient

Gradients growing as they propagate backwards until a step throws the weights far out of range, usually showing up as a loss that becomes NaN. Clipping and a lower learning rate are the usual answers.

A definition is not an understanding. This is taught properly in Vanishing and exploding gradients, measured, part of Backpropagation.

Learn this properly

Terms from the same topic

Cross-entropy loss AdamW Batch size Epoch Backpropagation Autograd Loss function Weight decay