The techniques, and what each establishes

Interpretability has four workhorse methods, and the useful thing to know about them is not how each is implemented but what a result from each one entitles you to claim. This section sets them side by side: what attribution shows, why a probe finding information somewhere does not mean the model uses it, what makes patching the one that establishes a dependency, and what a sparse autoencoder is attempting. Most disputed interpretability claims come from reading a correlational result as a causal one.

Pick the method from the sentence you want to write#

The four methods answer four different questions, and choosing between them is a matter of deciding in advance what you want to be able to say. Read the right-hand column as the strongest sentence a positive result licenses, and notice how much weaker three of them are than people write.

the question you are actually asking        the method     what a positive result licenses

which parts of the input moved the output   attribution    "these input positions influenced
                                                            the output" - nothing about why

is X represented anywhere in this layer     probing        "X is linearly recoverable here"
                                                            - not that the model reads it

does the output depend on this location     patching       "changing this location changes
                                                            the answer" - a real dependency

what are the units this layer works in      SAE            "here is a basis that reconstructs
                                                            the layer sparsely" - a hypothesis

Attribution#

Which inputs influenced this output. Gradient-based methods measure how the output changes as each input is perturbed. It is cheap and it answers a narrow question: where the model looked, not what it concluded.

The simplest version is the gradient of the output logit with respect to each input embedding, sometimes multiplied by the embedding itself. It costs one backward pass, which is why it is the method most likely to be reached for and the one most likely to be over-read. There is a control that costs almost nothing and is worth running every time: randomise the model's weights layer by layer and recompute the map. If a saliency map over a randomised network still looks like a sensible highlight of the important words, the map is telling you about the input and the architecture rather than about anything the model learned. Attribution maps that fail that check are common.

Probing#

Train a small classifier on internal activations to test whether some property is linearly recoverable there. If it is, that information is present in the representation.

The problem is that presence is not use. A probe can recover a property the model never consults. A strong probe is also sometimes learning the property itself rather than reading it, which is why probe capacity has to be constrained.

Both of those failures are easier to believe once you have made them happen, so the rest of this heading is one probe built and then dismantled by its own controls. The model is a two-layer transformer trained on a task small enough to be certain what it computes. Each input is three tokens: an item, a second item, and a cue saying which of the two to answer about. Each item has a colour, which is what the task asks for. The model reaches perfect accuracy in about ten seconds on a laptop.

import torch, torch.nn as nn

torch.manual_seed(0)
N_ITEMS, D, N_HEADS = 48, 32, 4
CUE_FIRST, CUE_SECOND = N_ITEMS, N_ITEMS + 1

def colour(item): return item % 4      # the attribute the task is about

class Block(nn.Module):
    def __init__(self):
        super().__init__()
        self.attn = nn.MultiheadAttention(D, N_HEADS, batch_first=True)
        self.mlp = nn.Sequential(nn.Linear(D, 4 * D), nn.ReLU(), nn.Linear(4 * D, D))

    def forward(self, x):
        x = x + self.attn(x, x, x, need_weights=False)[0]
        return x + self.mlp(x)

class Toy(nn.Module):
    def __init__(self):
        super().__init__()
        self.embed = nn.Embedding(N_ITEMS + 2, D)
        self.pos = nn.Parameter(torch.randn(3, D) * 0.02)
        self.blocks = nn.ModuleList([Block(), Block()])
        self.out = nn.Linear(D, 4)

    def forward(self, tokens, cache=None):
        x = self.embed(tokens) + self.pos
        for i, blk in enumerate(self.blocks):
            x = blk(x)
            if cache is not None:
                cache[f"resid_{i}"] = x.detach()
        return self.out(x[:, -1])

def make_batch(n):
    """[item 1, item 2, cue] -> the colour of whichever item the cue names."""
    a = torch.randint(0, N_ITEMS, (n,)); b = torch.randint(0, N_ITEMS, (n,))
    cue = torch.randint(0, 2, (n,))
    toks = torch.stack([a, b, torch.where(cue == 0, CUE_FIRST, CUE_SECOND)], 1)
    return toks, torch.where(cue == 0, colour(a), colour(b))

model = Toy()
opt = torch.optim.Adam(model.parameters(), lr=3e-3)
for step in range(800):
    toks, labels = make_batch(256)
    loss = nn.functional.cross_entropy(model(toks), labels)
    opt.zero_grad(); loss.backward(); opt.step()
with torch.no_grad():
    t, l = make_batch(4000)
    print(f"held-out accuracy: {(model(t).argmax(-1) == l).float().mean():.3f}")
# held-out accuracy: 1.000

Now the question worth probing. Restrict to the runs where the cue says answer about item 1. The colour of item 2 is then irrelevant: the model is being told to ignore it. Does the representation at the final position carry it anyway? A probe is a logistic regression trained on those activations, scored on a held-out quarter of the runs.

import numpy as np
from sklearn.linear_model import LogisticRegression

toks, _ = make_batch(8000)
toks = toks[toks[:, 2] == CUE_FIRST]        # keep only runs where item 1 is the answer
answered = colour(toks[:, 0]).numpy()       # the colour the model outputs
ignored = colour(toks[:, 1]).numpy()        # the colour of the item it was told to skip

def final_resid(m):
    cache = {}
    with torch.no_grad(): m(toks, cache=cache)
    return cache["resid_1"][:, -1].numpy()  # last layer, last position

def probe(X, y, train, test):
    clf = LogisticRegression(max_iter=3000).fit(X[train], y[train])
    return clf.score(X[test], y[test])

X = final_resid(model)
n = len(X); tr, te = slice(0, n * 3 // 4), slice(n * 3 // 4, n)
print(f"{n} runs. Chance for a 4-way label is 0.250.")
print(f"probe for the colour it answered with: {probe(X, answered, tr, te):.3f}")
print(f"probe for the colour it ignored:       {probe(X, ignored, tr, te):.3f}")
# 3951 runs. Chance for a 4-way label is 0.250.
# probe for the colour it answered with: 1.000
# probe for the colour it ignored:       0.598

That looks like a result. The colour the model was told to ignore is recoverable at 0.598 against a chance rate of 0.250, more than twice chance, and the sentence writes itself: the model represents the distractor even when instructed not to use it. Three controls decide whether that sentence survives.

1. 1.An untrained network. Run the identical probe on a randomly initialised model. Whatever it scores is the amount you get from random mixing of the inputs, and it is not a claim about training.
2. 2.Shuffled labels. Permute the labels so no relationship can exist, and probe again. Anything above chance here is the probe fitting noise rather than reading the model.
3. 3.Held-out entities. Split so that items 36 to 47 appear only in the probe's test set. A probe that has memorised which embedding goes with which colour cannot transfer, so this separates reading a general feature from recognising specific tokens.

torch.manual_seed(7)
Xu = final_resid(Toy())                     # control 1: a network that learnt nothing
rng = np.random.default_rng(0)
shuffled = rng.permutation(ignored)         # control 2: labels that cannot be there
held = (toks[:, 1] >= 36).numpy()           # control 3: items the probe never trained on
tr2, te2 = np.flatnonzero(~held), np.flatnonzero(held)

print("                                            trained   untrained")
print(f"random split,   colour it answered with     {probe(X, answered, tr, te):.3f}     {probe(Xu, answered, tr, te):.3f}")
print(f"random split,   colour it ignored           {probe(X, ignored, tr, te):.3f}     {probe(Xu, ignored, tr, te):.3f}")
print(f"held-out items, colour it answered with     {probe(X, answered, tr2, te2):.3f}     {probe(Xu, answered, tr2, te2):.3f}")
print(f"held-out items, colour it ignored           {probe(X, ignored, tr2, te2):.3f}     {probe(Xu, ignored, tr2, te2):.3f}")
print(f"shuffled labels on the trained model:       {probe(X, shuffled, tr, te):.3f}")
#                                             trained   untrained
# random split,   colour it answered with     1.000     0.566
# random split,   colour it ignored           0.598     0.572
# held-out items, colour it answered with     1.000     0.464
# held-out items, colour it ignored           0.513     0.224
# shuffled labels on the trained model:       0.234

Read the table row by row. The colour the model answers with probes at 1.000 against an untrained baseline of 0.566, which is a real and enormous effect, as it must be, since the model outputs it. The ignored colour probes at 0.598 against an untrained baseline of 0.572. The headline number was almost entirely random mixing, and the claim evaporates.

The held-out rows then flip the conclusion a second time. With items 36 to 47 excluded from the probe's training set, the ignored colour still probes at 0.513 while the untrained model drops to 0.224, below chance, because its memorised rules actively mislead on unseen items. So there is something real: training made colour linearly readable in a way that generalises to items the probe never saw. It is a different, smaller and better-supported claim than the one the first number suggested, and you only reach it by running both controls.

The capacity problem is the same story at a smaller scale. A linear probe on 32 dimensions fits 40 rows of pure noise perfectly, and the training accuracy tells you nothing.

from sklearn.linear_model import LogisticRegression as LR
for n_train in (40, 200, 2000):
    clf = LR(max_iter=5000, C=1e4).fit(X[:n_train], shuffled[:n_train])
    print(f"shuffled labels, {n_train:4d} training rows: "
          f"train {clf.score(X[:n_train], shuffled[:n_train]):.3f}  "
          f"test {clf.score(X[te], shuffled[te]):.3f}")
# shuffled labels,   40 training rows: train 1.000  test 0.236
# shuffled labels,  200 training rows: train 0.460  test 0.243
# shuffled labels, 2000 training rows: train 0.326  test 0.257

Even the surviving claim is about recoverability. Nothing above shows the model uses the ignored colour for anything, and in this task it provably does not, because changing item 2 while the cue says FIRST leaves the answer identical. Showing that a representation is used takes an intervention, which is the next method.

Activation patching#

This is the one that establishes causation. Run the model on two inputs, take the activation at one location from one run, insert it into the other, and see whether the output changes. If it does, that location carries information the output depends on.

clean run:      "The capital of France is" -> Paris
corrupted run:  "The capital of Italy is"  -> Rome

Patch one attention head's output from clean into corrupted.
If the answer flips to Paris, that head carries the country.

This is causal. Probing would only have shown the
information was present somewhere.

Three choices decide whether a patching result means anything, and all three are made before you run it. The corruption: a minimal edit that flips the answer, since a corruption that changes several things at once finds several things at once. The metric: the difference between the two answer logits, because accuracy is too coarse to register partial restoration and probability saturates near one. And the direction: patching clean activations into a corrupted run measures what is sufficient to restore the behaviour, while patching corrupted into clean measures what is necessary to destroy it, and the two can disagree when components back each other up. Say which one you ran.

Sparse autoencoders#

The response to superposition. Train an autoencoder with far more hidden units than the layer has dimensions, and a sparsity penalty so few activate at once. The hope is that it separates overlapping features into individually interpretable directions.

Two numbers describe any trained SAE and should be quoted together, because each alone can be made arbitrarily good at the other's expense. The reconstruction error says how much of the layer survives the round trip. The average number of features active per token, usually written L0, says how sparse the code is. A penalty weight of zero gives perfect reconstruction and a dense, uninterpretable code; a large penalty gives a beautifully sparse code that has thrown away the activation. Anyone reporting an SAE without both numbers has reported half of it.

The third number is the share of features that stop firing on anything and never recover, and it is routinely a third or more of the dictionary. Those are capacity you paid for and did not get.

It has produced genuinely interpretable features, and whether they are the model's features or artefacts of the autoencoder is an active question. Treat published feature interpretations as promising rather than settled.

What to carry into an interview#

• The four-row table above, as a sentence each. An interviewer asking about interpretability methods is usually testing whether you distinguish correlational from causal evidence, and that table is the answer.
• That a probe needs three controls named without prompting: an untrained model, shuffled labels, and a split that holds out entities. Volunteering them is the difference between a passable and a strong answer.
• That a patching result belongs to the prompt pair it was run on until replicated across many.
• Skip memorising SAE architecture variants. Knowing that reconstruction, L0 and the dead share must be reported together is worth more than knowing four names for the same encoder.