Asked about How training actually works

Your loss is not decreasing. How do you debug it?

An answer that works

First check the model can overfit a single batch, which isolates whether the setup works at all. Then the learning rate, which dominates everything. Then the data: labels correct, inputs scaled, no NaNs. Then whether zero_grad is being called.

Reading an answer is not being able to give one. The section that teaches this is How training actually works, 5 hours of Backpropagation.

Learn this properly

Also asked about Backpropagation

What does Adam do that SGD does not?Why use a learning rate warmup?What happens if you forget optimizer.zero_grad()?How does batch size relate to learning rate?Why AdamW rather than Adam with L2 regularisation?You have written a custom layer. How do you know its backward pass is right?How does reverse-mode automatic differentiation work?What causes vanishing gradients and how are they addressed?