Asked about Activation patching, in code

Walk me through an activation patching experiment.

An answer that works

Choose a behaviour and a clean prompt, make a corrupted prompt that changes the answer by a minimal edit, pick a metric such as the logit difference between the two answers, then copy activations from the clean run into the corrupted run one location at a time and measure how much of the clean behaviour each restores.

Then they push further

Why is this stronger evidence than probing?

A probe shows information can be decoded from a location. Patching intervenes: if replacing that activation changes the output, the model's computation actually depends on it there.

What could make a patching result misleading?

An unrepresentative corruption, a metric that saturates, one prompt standing in for a whole behaviour, and backup components that take over when one is changed. Replicate across many prompt pairs and corruptions.

What gets marked up

Strong answers explain clean, corrupted and metric, stress the causal nature, and volunteer the limitations.

Reading an answer is not being able to give one. The section that teaches this is Activation patching, in code, 5 hours of Interpretability.

Learn this properly

Also asked about Interpretability

What is mechanistic interpretability?What is superposition and why does it make interpretability hard?What can interpretability actually establish today?What is the difference between probing and activation patching?What is superposition and what is the response to it?Why is attribution not an explanation?What is the residual stream, and why is it a useful way to think about a transformer?How would you check whether an SAE feature really means what its label says?

All 10 questions on Interpretability