Backpropagation interview questions

Your loss is not decreasing. How do you debug it?

First check the model can overfit a single batch, which isolates whether the setup works at all. Then the learning rate, which dominates everything. Then the data: labels correct, inputs scaled, no NaNs. Then whether zero_grad is being called.

What does Adam do that SGD does not?

It adapts the step size per parameter using running estimates of the gradient and its square, so parameters with consistently small gradients still move. It is less sensitive to the initial learning rate, which is why it is the usual default.

Why use a learning rate warmup?

Early in training the weights are random and gradients can be large, so a full-size step can destabilise the run. Warming up from a small rate avoids that, and it matters most for large models and large batches.

What happens if you forget optimizer.zero_grad()?

Gradients accumulate across batches instead of being replaced, so each update is the sum of all previous gradients. Updates grow and training diverges, with no error raised.

How does batch size relate to learning rate?

Larger batches give less noisy gradients, which tolerate a larger learning rate, so the two are often scaled together. Changing one alone can destabilise or slow training.

Why AdamW rather than Adam with L2 regularisation?

Adam rescales gradients per parameter, so an L2 term in the loss does not act like true weight decay. AdamW applies decay directly to the weights, decoupled from the gradient scaling.

You have written a custom layer. How do you know its backward pass is right?

A gradient check. Compute the analytic gradient, then for each parameter nudge it by about 1e-5 in both directions, recompute the loss, and divide the difference by twice the step. Compare the two as a relative error rather than an absolute difference: below 1e-7 is correct, above 1e-4 is a bug. Run it on the smallest input that exercises the layer, because it costs two forward passes per parameter.

What gets marked up. Marked up for relative rather than absolute comparison, for knowing the step size has a lower limit, and for saying it runs once on a tiny model rather than in the training loop.

How does reverse-mode automatic differentiation work?

During the forward pass each operation records its inputs and how to compute its local derivative. Backward starts with a gradient of 1 at the output and visits nodes in reverse topological order, each passing its gradient times its local derivative to its inputs, summing contributions where a value was used more than once.

What gets marked up. Marked up for topological order, summing gradients, and the reverse versus forward mode argument.

What causes vanishing gradients and how are they addressed?

Backpropagation multiplies local derivatives layer by layer; if they are consistently below one, as with saturating activations or poorly scaled weights, gradients shrink exponentially towards early layers. Fixes are ReLU-family activations, careful initialisation, normalisation layers and residual connections.