Asked about A neural network from scratch in NumPy

Walk me through backpropagation for a two-layer network.

An answer that works

Run the forward pass, keeping intermediate values. The gradient of the loss with respect to the logits is the probabilities minus the targets. From there, the output weight gradient is the hidden activations transposed times that; the hidden layer's gradient is that error sent back through the output weights, zeroed where the ReLU was inactive; and the first layer's weight gradient is the input transposed times that.

Then they push further

How do you verify a hand-written backward pass?

A gradient check: nudge each parameter up and down by a tiny amount, compute the change in loss numerically, and compare with the analytic gradient. The relative difference should be tiny.

Why initialise weights with small random values rather than zeros?

With all-zero weights every hidden unit computes the same thing and receives the same gradient, so they never become different. Random values break the symmetry; their scale must be chosen so signals neither vanish nor explode through the layers.

What gets marked up

Implementing this live, or explaining it with shapes, is a standard research and ML engineer interview task.

Reading an answer is not being able to give one. The section that teaches this is A neural network from scratch in NumPy, 6 hours of Neural networks.

Learn this properly

Also asked about Neural networks

Why do neural networks need activation functions?Why is ReLU the default rather than sigmoid?What is the dying ReLU problem?Which loss would you use for multi-class classification?Why do residual connections help deep networks?Why do transformers use layer norm rather than batch norm?Your validation loss starts rising while training loss keeps falling. What is happening and what do you do?