Asked about How training actually works
Why use a learning rate warmup?
An answer that works
Early in training the weights are random and gradients can be large, so a full-size step can destabilise the run. Warming up from a small rate avoids that, and it matters most for large models and large batches.
Reading an answer is not being able to give one. The section that teaches this is How training actually works, 5 hours of Backpropagation.
Learn this properlyAlso asked about Backpropagation
Your loss is not decreasing. How do you debug it?What does Adam do that SGD does not?What happens if you forget optimizer.zero_grad()?How does batch size relate to learning rate?Why AdamW rather than Adam with L2 regularisation?You have written a custom layer. How do you know its backward pass is right?How does reverse-mode automatic differentiation work?What causes vanishing gradients and how are they addressed?