Asked about How training actually works
What happens if you forget optimizer.zero_grad()?
An answer that works
Gradients accumulate across batches instead of being replaced, so each update is the sum of all previous gradients. Updates grow and training diverges, with no error raised.
Reading an answer is not being able to give one. The section that teaches this is How training actually works, 5 hours of Backpropagation.
Learn this properlyAlso asked about Backpropagation
Your loss is not decreasing. How do you debug it?What does Adam do that SGD does not?Why use a learning rate warmup?How does batch size relate to learning rate?Why AdamW rather than Adam with L2 regularisation?You have written a custom layer. How do you know its backward pass is right?How does reverse-mode automatic differentiation work?What causes vanishing gradients and how are they addressed?