Asked about Derivatives and the chain rule
Why do gradients vanish in deep networks?
An answer that works
The chain rule multiplies derivatives along the path, so many factors below one shrink towards zero and early layers receive almost no signal. Residual connections and activations like ReLU exist largely to counter this.
Reading an answer is not being able to give one. The section that teaches this is Derivatives and the chain rule, 5 hours of Calculus for ML.
Learn this properlyAlso asked about Calculus for ML
What does the learning rate actually do, and how do you tell it is wrong?What is a gradient?Your loss becomes NaN during training. What do you check?Are local minima the main problem in training deep networks?What does momentum do?What is the gradient of cross-entropy loss with softmax, with respect to the logits?Explain the Adam update.