Adam

An optimiser keeping a running average of each parameter's gradient and of its squared gradient, and dividing one by the other. Parameters with small, consistent gradients get large steps and vice versa, which makes it forgiving about learning rate.

A definition is not an understanding. This is taught properly in Gradient descent, momentum and Adam from scratch, part of Calculus for ML.

Learn this properly

Terms from the same topic

Gradient descent Stochastic gradient descent Gradient clipping Gradient Partial derivative Chain rule