Neural networks interview questions

Why do neural networks need activation functions?

Without them, stacked linear layers collapse into a single linear transformation, since a matrix multiply of matrix multiplies is one matrix multiply. The non-linearity is what makes depth add expressive power.

Why is ReLU the default rather than sigmoid?

Sigmoid saturates at both ends, so gradients approach zero and early layers stop learning. ReLU does not saturate for positive inputs, keeping gradients alive, and it is cheaper to compute.

What is the dying ReLU problem?

A unit whose input is always negative outputs zero and has zero gradient, so it never updates and is permanently dead. Leaky ReLU and GELU avoid it by keeping a small gradient for negative inputs.

Which loss would you use for multi-class classification?

Cross-entropy, usually with softmax at the output. It penalises confident wrong predictions heavily, which is the behaviour you want, whereas squared error treats a confidently wrong answer too gently.

Why do residual connections help deep networks?

They add each layer's input to its output, giving gradients a direct path back through the network. Without them gradients shrink through each layer's transformation; with them even early layers receive a usable signal.

Why do transformers use layer norm rather than batch norm?

Layer norm normalises across each example's features independently, so it does not depend on batch composition or size, which suits variable-length sequences. Batch norm's reliance on batch statistics is awkward there.

Walk me through backpropagation for a two-layer network.

Run the forward pass, keeping intermediate values. The gradient of the loss with respect to the logits is the probabilities minus the targets. From there, the output weight gradient is the hidden activations transposed times that; the hidden layer's gradient is that error sent back through the output weights, zeroed where the ReLU was inactive; and the first layer's weight gradient is the input transposed times that.

What gets marked up. Implementing this live, or explaining it with shapes, is a standard research and ML engineer interview task.

Your validation loss starts rising while training loss keeps falling. What is happening and what do you do?

The model is overfitting: memorising training examples rather than learning patterns that generalise. Stop at the best validation point, then add regularisation such as weight decay, dropout or augmentation, get more data, or reduce model size.