Asked about How training actually works
What does Adam do that SGD does not?
An answer that works
It adapts the step size per parameter using running estimates of the gradient and its square, so parameters with consistently small gradients still move. It is less sensitive to the initial learning rate, which is why it is the usual default.
Reading an answer is not being able to give one. The section that teaches this is How training actually works, 5 hours of Backpropagation.
Learn this properlyAlso asked about Backpropagation
Your loss is not decreasing. How do you debug it?Why use a learning rate warmup?What happens if you forget optimizer.zero_grad()?How does batch size relate to learning rate?Why AdamW rather than Adam with L2 regularisation?You have written a custom layer. How do you know its backward pass is right?How does reverse-mode automatic differentiation work?What causes vanishing gradients and how are they addressed?