Asked about Derivatives and the chain rule
Your loss becomes NaN during training. What do you check?
An answer that works
Exploding gradients first: clip them and lower the learning rate. Then check for division by zero or a log of zero in the loss, and for bad input data such as NaNs already present in the batch.
Reading an answer is not being able to give one. The section that teaches this is Derivatives and the chain rule, 5 hours of Calculus for ML.
Learn this properlyAlso asked about Calculus for ML
What does the learning rate actually do, and how do you tell it is wrong?What is a gradient?Why do gradients vanish in deep networks?Are local minima the main problem in training deep networks?What does momentum do?What is the gradient of cross-entropy loss with softmax, with respect to the logits?Explain the Adam update.