Asked about Derivatives and the chain rule

What does the learning rate actually do, and how do you tell it is wrong?

An answer that works

The gradient gives a direction; the learning rate is how far you step along it. There is a threshold set by the curvature of the loss: on a loss of 5 w squared the update multiplies w by 1 minus ten times the rate, so above 0.2 the multiplier exceeds one in size and the run diverges. The symptoms are diagnostic. Oscillation with growing amplitude means above the threshold; oscillation with steady amplitude means close to it; a smooth but barely moving loss means far below it. The threshold depends on the problem, which is why a rate copied from another project usually fails.

Then they push further

Your loss falls smoothly and the curve looks healthy. Can the rate still be wrong?

Yes, and that is the expensive case. A rate a hundred times too small gives a curve that looks entirely normal and an answer that is simply unfinished. The only way to see it is a comparison: a longer run, a larger rate, or a closed-form solution where one exists.

Reading an answer is not being able to give one. The section that teaches this is Derivatives and the chain rule, 5 hours of Calculus for ML.

Learn this properly

Also asked about Calculus for ML

What is a gradient?Why do gradients vanish in deep networks?Your loss becomes NaN during training. What do you check?Are local minima the main problem in training deep networks?What does momentum do?What is the gradient of cross-entropy loss with softmax, with respect to the logits?Explain the Adam update.