Asked about Vanishing and exploding gradients, measured
What causes vanishing gradients and how are they addressed?
An answer that works
Backpropagation multiplies local derivatives layer by layer; if they are consistently below one, as with saturating activations or poorly scaled weights, gradients shrink exponentially towards early layers. Fixes are ReLU-family activations, careful initialisation, normalisation layers and residual connections.
Reading an answer is not being able to give one. The section that teaches this is Vanishing and exploding gradients, measured, 3 hours of Backpropagation.
Learn this properlyAlso asked about Backpropagation
Your loss is not decreasing. How do you debug it?What does Adam do that SGD does not?Why use a learning rate warmup?What happens if you forget optimizer.zero_grad()?How does batch size relate to learning rate?Why AdamW rather than Adam with L2 regularisation?You have written a custom layer. How do you know its backward pass is right?How does reverse-mode automatic differentiation work?