Residual stream

The running sum carried from the bottom of a transformer to the top, which every attention and feed-forward layer reads from and writes into. Treating it as a shared bus rather than a pipeline is what makes most interpretability work possible.

A definition is not an understanding. This is taught properly in Looking inside a transformer with TransformerLens, part of Interpretability.

Learn this properly