Layer normalisation
Also: layer norm, LayerNorm
Rescaling each vector so its values have zero mean and unit variance across the feature dimension, then applying a learned scale and shift. It keeps activations in a range where gradients stay usable as the stack deepens.
What people get wrong
Unlike batch normalisation it looks at one example at a time, so batch size does not change its behaviour. That is why sequence models use it and vision models historically did not.
A definition is not an understanding. This is taught properly in Initialisation, normalisation and residuals, part of Neural networks.
Learn this properlyTerms from the same topic