Asked about Looking inside a transformer with TransformerLens
What is the residual stream, and why is it a useful way to think about a transformer?
An answer that works
The per-position vector that runs from embedding to output, which each attention and MLP layer reads from and adds to. Because contributions are added rather than overwritten, you can attribute the final output to individual components and ask what each one contributed.
Then they push further
What does the logit lens show, and what is its limitation?
It decodes intermediate residual states with the final unembedding to show the model's best guess at each layer. It assumes intermediate states live in the same space as the final one, which holds only roughly, especially in early layers.
How would you find an induction head?
Feed a random token sequence repeated twice and look for heads that, in the second half, attend strongly to the position just after the previous occurrence of the current token. Then confirm causally by ablating them and watching loss on the repeated half rise.
What gets marked up
Marked up for the additive view of the residual stream and for confirming a found circuit causally, not just by looking at attention.
Reading an answer is not being able to give one. The section that teaches this is Looking inside a transformer with TransformerLens, 5 hours of Interpretability.
Learn this properlyAlso asked about Interpretability