Asked about Initialisation, normalisation and residuals
Why do residual connections help deep networks?
An answer that works
They add each layer's input to its output, giving gradients a direct path back through the network. Without them gradients shrink through each layer's transformation; with them even early layers receive a usable signal.
Reading an answer is not being able to give one. The section that teaches this is Initialisation, normalisation and residuals, 4 hours of Neural networks.
Learn this properlyAlso asked about Neural networks
Why do neural networks need activation functions?Why is ReLU the default rather than sigmoid?What is the dying ReLU problem?Which loss would you use for multi-class classification?Why do transformers use layer norm rather than batch norm?Walk me through backpropagation for a two-layer network.Your validation loss starts rising while training loss keeps falling. What is happening and what do you do?