Asked about Automatic differentiation from scratch

You have written a custom layer. How do you know its backward pass is right?

An answer that works

A gradient check. Compute the analytic gradient, then for each parameter nudge it by about 1e-5 in both directions, recompute the loss, and divide the difference by twice the step. Compare the two as a relative error rather than an absolute difference: below 1e-7 is correct, above 1e-4 is a bug. Run it on the smallest input that exercises the layer, because it costs two forward passes per parameter.

Then they push further

Why not use a much smaller step for the numerical gradient?

Below about 1e-8 the two loss values are so close that floating point subtraction throws away the digits carrying the answer. At 1e-15 the estimate is wrong in the first decimal place. There is a window, and 1e-5 sits in it.

What kinds of bug does this actually catch?

A missing activation mask, a reduction over the wrong axis, a sign error, and a factor left out. In my experience the mask is the most common, because the forward pass still looks correct and the model still trains, just worse than it should.

What gets marked up

Marked up for relative rather than absolute comparison, for knowing the step size has a lower limit, and for saying it runs once on a tiny model rather than in the training loop.

Reading an answer is not being able to give one. The section that teaches this is Automatic differentiation from scratch, 5 hours of Backpropagation.

Learn this properly

Also asked about Backpropagation

Your loss is not decreasing. How do you debug it?What does Adam do that SGD does not?Why use a learning rate warmup?What happens if you forget optimizer.zero_grad()?How does batch size relate to learning rate?Why AdamW rather than Adam with L2 regularisation?How does reverse-mode automatic differentiation work?What causes vanishing gradients and how are they addressed?