Gradient descent

Repeatedly nudging every parameter a small step in the direction that most reduces the loss. Nearly all model training is some refinement of this one idea.

A definition is not an understanding. This is taught properly in Gradient descent, momentum and Adam from scratch, part of Calculus for ML.

Learn this properly

Terms from the same topic

Stochastic gradient descent Adam Gradient clipping Gradient Partial derivative Chain rule