AdamW

Adam with weight decay applied directly to the weights instead of folded into the gradient. The default optimiser for transformers, because the original coupling made decay effectively vanish for exactly the parameters it should shrink.

A definition is not an understanding. This is taught properly in Batch size, schedules and regularisation in practice, part of Backpropagation.

Learn this properly

Terms from the same topic

Cross-entropy loss Batch size Epoch Backpropagation Autograd Loss function Weight decay Vanishing gradient