Learning rate schedule

A plan for changing the learning rate over a run, usually a warmup followed by a slow decay to near zero. Large steps early to make progress, small steps late to settle.

A definition is not an understanding. This is taught properly in Learning rate schedules and a tuning workflow, part of Hyperparameter tuning.

Learn this properly

Terms from the same topic