Calculus for ML interview questions
8 questions across 4 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Derivatives and the chain rule
Read the section that teaches thisIf this comes up in an interview
What does the learning rate actually do, and how do you tell it is wrong?
The gradient gives a direction; the learning rate is how far you step along it. There is a threshold set by the curvature of the loss: on a loss of 5 w squared the update multiplies w by 1 minus ten times the rate, so above 0.2 the multiplier exceeds one in size and the run diverges. The symptoms are diagnostic. Oscillation with growing amplitude means above the threshold; oscillation with steady amplitude means close to it; a smooth but barely moving loss means far below it. The threshold depends on the problem, which is why a rate copied from another project usually fails.
Then they push further
Your loss falls smoothly and the curve looks healthy. Can the rate still be wrong?
Yes, and that is the expensive case. A rate a hundred times too small gives a curve that looks entirely normal and an answer that is simply unfinished. The only way to see it is a comparison: a longer run, a larger rate, or a closed-form solution where one exists.
What is a gradient?
The vector of partial derivatives of the loss with respect to every parameter. It points in the direction of steepest increase, so training steps against it to reduce the loss.
Why do gradients vanish in deep networks?
The chain rule multiplies derivatives along the path, so many factors below one shrink towards zero and early layers receive almost no signal. Residual connections and activations like ReLU exist largely to counter this.
Your loss becomes NaN during training. What do you check?
Exploding gradients first: clip them and lower the learning rate. Then check for division by zero or a log of zero in the loss, and for bad input data such as NaNs already present in the batch.
The shape of the loss surface
Read the section that teaches thisIf this comes up in an interview
Are local minima the main problem in training deep networks?
Mostly not. In high dimensions, most flat points are saddle points, where some directions still go downhill. Saddle points and plateaus slow training far more often than true local minima trap it.
What does momentum do?
It accumulates a running average of past gradients, so consistent directions gain speed and oscillations cancel. That carries the optimiser across saddle points and plateaus and damps zigzagging in narrow valleys.
Derivatives you can check in code
Read the section that teaches thisIf this comes up in an interview
What is the gradient of cross-entropy loss with softmax, with respect to the logits?
The predicted probability vector minus the one-hot target vector. It is simple and well-behaved, which is one reason softmax and cross-entropy are always used together.
Then they push further
Why is sigmoid a poor activation for hidden layers in deep networks?
Its derivative is at most 0.25 and near zero for large inputs, so multiplying many of them in the chain rule shrinks gradients towards zero in early layers. ReLU-family activations have a derivative of 1 for positive inputs.
What gets marked up. Marked up for p minus y without hesitation and for linking the sigmoid derivative to vanishing gradients.
Gradient descent, momentum and Adam from scratch
Read the section that teaches thisIf this comes up in an interview
Explain the Adam update.
It keeps a running average of gradients, which acts as momentum, and a running average of squared gradients, which estimates each parameter's gradient scale. Both are bias-corrected for the early steps. The update is the averaged gradient divided by the square root of the averaged squared gradient, times the learning rate.
Then they push further
What does AdamW change?
It applies weight decay directly to the weights rather than adding it to the gradient, where Adam's scaling would distort it. That makes regularisation behave as intended.
Why does Adam need so much memory compared with SGD?
It stores two extra values per parameter, the two running averages, typically in 32-bit precision. For large models the optimiser state is bigger than the weights.
What gets marked up. Strong answers explain both moments, the bias correction, and connect memory cost to training large models.
These are the questions for one topic. The whole question bank covers every topic on the site.