Asked about Gradient descent, momentum and Adam from scratch
Explain the Adam update.
An answer that works
It keeps a running average of gradients, which acts as momentum, and a running average of squared gradients, which estimates each parameter's gradient scale. Both are bias-corrected for the early steps. The update is the averaged gradient divided by the square root of the averaged squared gradient, times the learning rate.
Then they push further
What does AdamW change?
It applies weight decay directly to the weights rather than adding it to the gradient, where Adam's scaling would distort it. That makes regularisation behave as intended.
Why does Adam need so much memory compared with SGD?
It stores two extra values per parameter, the two running averages, typically in 32-bit precision. For large models the optimiser state is bigger than the weights.
What gets marked up
Strong answers explain both moments, the bias correction, and connect memory cost to training large models.
Reading an answer is not being able to give one. The section that teaches this is Gradient descent, momentum and Adam from scratch, 4 hours of Calculus for ML.
Learn this properlyAlso asked about Calculus for ML