Asked about Batch size, schedules and regularisation in practice

Why AdamW rather than Adam with L2 regularisation?

An answer that works

Adam rescales gradients per parameter, so an L2 term in the loss does not act like true weight decay. AdamW applies decay directly to the weights, decoupled from the gradient scaling.

Reading an answer is not being able to give one. The section that teaches this is Batch size, schedules and regularisation in practice, 4 hours of Backpropagation.

Learn this properly

Also asked about Backpropagation

Your loss is not decreasing. How do you debug it?What does Adam do that SGD does not?Why use a learning rate warmup?What happens if you forget optimizer.zero_grad()?How does batch size relate to learning rate?You have written a custom layer. How do you know its backward pass is right?How does reverse-mode automatic differentiation work?What causes vanishing gradients and how are they addressed?