Asked about Initialisation, normalisation and residuals
Why do transformers use layer norm rather than batch norm?
An answer that works
Layer norm normalises across each example's features independently, so it does not depend on batch composition or size, which suits variable-length sequences. Batch norm's reliance on batch statistics is awkward there.
Reading an answer is not being able to give one. The section that teaches this is Initialisation, normalisation and residuals, 4 hours of Neural networks.
Learn this properlyAlso asked about Neural networks
Why do neural networks need activation functions?Why is ReLU the default rather than sigmoid?What is the dying ReLU problem?Which loss would you use for multi-class classification?Why do residual connections help deep networks?Walk me through backpropagation for a two-layer network.Your validation loss starts rising while training loss keeps falling. What is happening and what do you do?