Interpretability

We can tell that a model performs well, and not why. That gap matters whenever you need to trust it, debug it, or predict whether it will hold up on inputs unlike the ones you tested. Interpretability is the attempt to close it. This section covers the main approaches, from probing what information a representation carries to tracing the specific circuits that implement a behaviour, and is straight about what each can actually establish.

The problem#

We can measure that a model performs well and not why. That matters for trust, for debugging, for safety, and for knowing whether performance will hold on inputs unlike the test set.

What there is to look at#

Every technique here operates on activations, so it is worth being exact about what one is. A forward pass produces, at each layer and each token position, a vector of numbers as wide as the model. GPT-2 small is 12 layers and 768 dimensions, so a 20-token prompt leaves roughly 184,000 numbers behind at the layer boundaries alone, before counting anything inside attention or the MLPs. That pile is the whole subject matter. The job is to turn some of those numbers into a statement about the model that another person can check.

A feature is a direction, not a neuron#

The idea that makes the rest of the field workable is that the model's units of meaning are directions in activation space rather than individual coordinates. A direction is a unit vector as wide as the activation. The model writes a feature into the stream by adding a multiple of its direction, and reads it back by taking a dot product against a readout direction. Nothing more exotic is happening, and the whole mechanism fits in a few lines.

import torch

torch.manual_seed(0)
N_FEATURES, D_HIDDEN = 20, 5

W = torch.randn(N_FEATURES, D_HIDDEN)
W = W / W.norm(dim=1, keepdim=True)        # one unit direction per feature

x = torch.zeros(4, N_FEATURES)
x[0, 3] = 1.0                              # example 0: feature 3 on, alone
x[1, 7] = 1.0                              # example 1: feature 7 on, alone
x[2, 3] = 1.0; x[2, 7] = 0.5               # example 2: both, 7 at half strength
h = x @ W                                  # the activation: five numbers

print("activation for example 2:", [round(v, 2) for v in h[2].tolist()])
print("read feature 3 back:", [round(v, 2) for v in (h @ W[3]).tolist()])
print("read feature 7 back:", [round(v, 2) for v in (h @ W[7]).tolist()])
# activation for example 2: [0.03, -0.33, -0.65, 0.58, 0.57]
# read feature 3 back: [1.0, -0.06, 0.97, 0.0]
# read feature 7 back: [-0.06, 1.0, 0.44, 0.0]

Two consequences follow, and they set up everything else. A neuron is one coordinate, which is one particular direction out of infinitely many, and the model has no reason to prefer the coordinate axes; so asking what neuron 402 means is usually the wrong question. And because directions need not be orthogonal, features interfere, which is the -0.06 above. With few features relative to the width of the layer, that leakage is negligible. It is not negligible in a real model, and the reason is the next heading.

Superposition: more features than dimensions#

A central difficulty. Models represent more features than they have dimensions by packing them into overlapping directions, so a single neuron responds to several unrelated things. This is why reading individual neurons rarely works, and why sparse autoencoders became a major tool for separating features back out.

The reason the model gets away with it is sparsity. If most features are off in any given input, two features sharing an almost-identical direction rarely collide, and the interference costs less than not representing one of them at all. That is a claim you can test rather than accept, by squeezing twenty features through five dimensions at different sparsity levels and counting how many survive.

class ToyModel(torch.nn.Module):
    """Squeeze N features through D dimensions and try to get them all back."""
    def __init__(self):
        super().__init__()
        self.W = torch.nn.Parameter(torch.randn(N_FEATURES, D_HIDDEN) * 0.1)
        self.b = torch.nn.Parameter(torch.zeros(N_FEATURES))

    def forward(self, x):
        h = x @ self.W                                  # N -> D, the bottleneck
        return torch.relu(h @ self.W.T + self.b)        # D -> N, tied weights

def batch(sparsity, n=1024):
    magnitude = torch.rand(n, N_FEATURES)
    on = (torch.rand(n, N_FEATURES) < (1 - sparsity)).float()
    return magnitude * on

def train(sparsity, steps=4000):
    torch.manual_seed(0)
    m = ToyModel()
    opt = torch.optim.Adam(m.parameters(), lr=1e-2)
    importance = 0.9 ** torch.arange(N_FEATURES)        # earlier features matter more
    for _ in range(steps):
        x = batch(sparsity)
        loss = (importance * (m(x) - x) ** 2).mean()
        opt.zero_grad(); loss.backward(); opt.step()
    return m

for sparsity in (0.0, 0.7, 0.9, 0.99):
    m = train(sparsity)
    norms = m.W.norm(dim=1)
    live = norms > 0.5                                  # which features got a direction
    U = m.W / (norms[:, None] + 1e-8)
    cos = (U @ U.T).abs(); cos.fill_diagonal_(0)
    interference = cos[live][:, live].mean().item() if live.sum() > 1 else 0.0
    print(f"sparsity {sparsity:.2f}  represented {live.sum().item():2d}/20  "
          f"mean |cos| between them {interference:.2f}")
# sparsity 0.00  represented  5/20  mean |cos| between them 0.00
# sparsity 0.70  represented 10/20  mean |cos| between them 0.10
# sparsity 0.90  represented 12/20  mean |cos| between them 0.20
# sparsity 0.99  represented 18/20  mean |cos| between them 0.33

Read the two columns together. The dense row is the behaviour you would have predicted without ever hearing the word superposition: five dimensions, five features, no interference. Every row below it stores more features than the layer has dimensions, and the interference column is the price. A real layer sits at the bottom of that table, not the top, which is why a neuron in a trained network responds to a face, a piece of Python and the word Tuesday, and why nothing you can do to that neuron alone will separate them.

Approaches, roughly by depth#

• Attribution: which inputs mattered for this output. Useful and limited, since it says where the model looked and not what it concluded.
• Probing: train a small classifier on internal activations to test whether some property is represented there. Shows information is present, not that the model uses it.
• Activation patching: copy an activation from one run into another and see whether the output changes. The only one of the four that establishes a dependency rather than a correlation.
• Mechanistic interpretability: reverse-engineer the actual computation, identifying circuits that implement a behaviour. The most ambitious and the most informative when it works.

The useful way to hold these is by what a positive result entitles you to say. A probe that succeeds says the information is recoverable from that location by a linear map. A patch that changes the output says the model's computation at that location feeds the answer. Those are different claims, and the gap between them is where most disputed interpretability results live. Choose the method by the sentence you want to be able to write at the end, not by which one is easiest to run.

Honest about the state#

It is a real hiring area at the safety-focused labs, and one of the few where careful empirical work on small models still produces publishable results. It is also one of the few where an interviewer can tell within two questions whether you have run anything, because the answers that come from reading about the field and the answers that come from training a toy model do not sound alike.

What to carry, and what to skip#

• Carry: a feature is a direction, writing is addition along it, reading is a dot product. Every technique in the topic is a variation on those three sentences.
• Carry: superposition, and specifically why sparsity is what makes it pay. Without the sparsity half you cannot answer the follow-up.
• Carry: the distinction between what a probe establishes and what a patch establishes. It is the most frequently asked interpretability question outside a specialist team.
• Skip, for now: the taxonomy of named circuits in published papers. Knowing that induction heads exist is worth a sentence; reciting the list without having found one is worth nothing.
• Skip: choosing between saliency-map variants. The differences between them matter far less than the fact that none of them is causal.

Interpretability has four workhorse methods, and the useful thing to know about them is not how each is implemented but what a result from each one entitles you to claim. This section sets them side by side: what attribution shows, why a probe finding information somewhere does not mean the model uses it, what makes patching the one that establishes a dependency, and what a sparse autoencoder is attempting. Most disputed interpretability claims come from reading a correlational result as a causal one.

Pick the method from the sentence you want to write#

The four methods answer four different questions, and choosing between them is a matter of deciding in advance what you want to be able to say. Read the right-hand column as the strongest sentence a positive result licenses, and notice how much weaker three of them are than people write.

the question you are actually asking        the method     what a positive result licenses

which parts of the input moved the output   attribution    "these input positions influenced
                                                            the output" - nothing about why

is X represented anywhere in this layer     probing        "X is linearly recoverable here"
                                                            - not that the model reads it

does the output depend on this location     patching       "changing this location changes
                                                            the answer" - a real dependency

what are the units this layer works in      SAE            "here is a basis that reconstructs
                                                            the layer sparsely" - a hypothesis

Attribution#

Which inputs influenced this output. Gradient-based methods measure how the output changes as each input is perturbed. It is cheap and it answers a narrow question: where the model looked, not what it concluded.

The simplest version is the gradient of the output logit with respect to each input embedding, sometimes multiplied by the embedding itself. It costs one backward pass, which is why it is the method most likely to be reached for and the one most likely to be over-read. There is a control that costs almost nothing and is worth running every time: randomise the model's weights layer by layer and recompute the map. If a saliency map over a randomised network still looks like a sensible highlight of the important words, the map is telling you about the input and the architecture rather than about anything the model learned. Attribution maps that fail that check are common.

Probing#

Train a small classifier on internal activations to test whether some property is linearly recoverable there. If it is, that information is present in the representation.

The problem is that presence is not use. A probe can recover a property the model never consults. A strong probe is also sometimes learning the property itself rather than reading it, which is why probe capacity has to be constrained.

Both of those failures are easier to believe once you have made them happen, so the rest of this heading is one probe built and then dismantled by its own controls. The model is a two-layer transformer trained on a task small enough to be certain what it computes. Each input is three tokens: an item, a second item, and a cue saying which of the two to answer about. Each item has a colour, which is what the task asks for. The model reaches perfect accuracy in about ten seconds on a laptop.

import torch, torch.nn as nn

torch.manual_seed(0)
N_ITEMS, D, N_HEADS = 48, 32, 4
CUE_FIRST, CUE_SECOND = N_ITEMS, N_ITEMS + 1

def colour(item): return item % 4      # the attribute the task is about

class Block(nn.Module):
    def __init__(self):
        super().__init__()
        self.attn = nn.MultiheadAttention(D, N_HEADS, batch_first=True)
        self.mlp = nn.Sequential(nn.Linear(D, 4 * D), nn.ReLU(), nn.Linear(4 * D, D))

    def forward(self, x):
        x = x + self.attn(x, x, x, need_weights=False)[0]
        return x + self.mlp(x)

class Toy(nn.Module):
    def __init__(self):
        super().__init__()
        self.embed = nn.Embedding(N_ITEMS + 2, D)
        self.pos = nn.Parameter(torch.randn(3, D) * 0.02)
        self.blocks = nn.ModuleList([Block(), Block()])
        self.out = nn.Linear(D, 4)

    def forward(self, tokens, cache=None):
        x = self.embed(tokens) + self.pos
        for i, blk in enumerate(self.blocks):
            x = blk(x)
            if cache is not None:
                cache[f"resid_{i}"] = x.detach()
        return self.out(x[:, -1])

def make_batch(n):
    """[item 1, item 2, cue] -> the colour of whichever item the cue names."""
    a = torch.randint(0, N_ITEMS, (n,)); b = torch.randint(0, N_ITEMS, (n,))
    cue = torch.randint(0, 2, (n,))
    toks = torch.stack([a, b, torch.where(cue == 0, CUE_FIRST, CUE_SECOND)], 1)
    return toks, torch.where(cue == 0, colour(a), colour(b))

model = Toy()
opt = torch.optim.Adam(model.parameters(), lr=3e-3)
for step in range(800):
    toks, labels = make_batch(256)
    loss = nn.functional.cross_entropy(model(toks), labels)
    opt.zero_grad(); loss.backward(); opt.step()
with torch.no_grad():
    t, l = make_batch(4000)
    print(f"held-out accuracy: {(model(t).argmax(-1) == l).float().mean():.3f}")
# held-out accuracy: 1.000

Now the question worth probing. Restrict to the runs where the cue says answer about item 1. The colour of item 2 is then irrelevant: the model is being told to ignore it. Does the representation at the final position carry it anyway? A probe is a logistic regression trained on those activations, scored on a held-out quarter of the runs.

import numpy as np
from sklearn.linear_model import LogisticRegression

toks, _ = make_batch(8000)
toks = toks[toks[:, 2] == CUE_FIRST]        # keep only runs where item 1 is the answer
answered = colour(toks[:, 0]).numpy()       # the colour the model outputs
ignored = colour(toks[:, 1]).numpy()        # the colour of the item it was told to skip

def final_resid(m):
    cache = {}
    with torch.no_grad(): m(toks, cache=cache)
    return cache["resid_1"][:, -1].numpy()  # last layer, last position

def probe(X, y, train, test):
    clf = LogisticRegression(max_iter=3000).fit(X[train], y[train])
    return clf.score(X[test], y[test])

X = final_resid(model)
n = len(X); tr, te = slice(0, n * 3 // 4), slice(n * 3 // 4, n)
print(f"{n} runs. Chance for a 4-way label is 0.250.")
print(f"probe for the colour it answered with: {probe(X, answered, tr, te):.3f}")
print(f"probe for the colour it ignored:       {probe(X, ignored, tr, te):.3f}")
# 3951 runs. Chance for a 4-way label is 0.250.
# probe for the colour it answered with: 1.000
# probe for the colour it ignored:       0.598

That looks like a result. The colour the model was told to ignore is recoverable at 0.598 against a chance rate of 0.250, more than twice chance, and the sentence writes itself: the model represents the distractor even when instructed not to use it. Three controls decide whether that sentence survives.

1. 1.An untrained network. Run the identical probe on a randomly initialised model. Whatever it scores is the amount you get from random mixing of the inputs, and it is not a claim about training.
2. 2.Shuffled labels. Permute the labels so no relationship can exist, and probe again. Anything above chance here is the probe fitting noise rather than reading the model.
3. 3.Held-out entities. Split so that items 36 to 47 appear only in the probe's test set. A probe that has memorised which embedding goes with which colour cannot transfer, so this separates reading a general feature from recognising specific tokens.

torch.manual_seed(7)
Xu = final_resid(Toy())                     # control 1: a network that learnt nothing
rng = np.random.default_rng(0)
shuffled = rng.permutation(ignored)         # control 2: labels that cannot be there
held = (toks[:, 1] >= 36).numpy()           # control 3: items the probe never trained on
tr2, te2 = np.flatnonzero(~held), np.flatnonzero(held)

print("                                            trained   untrained")
print(f"random split,   colour it answered with     {probe(X, answered, tr, te):.3f}     {probe(Xu, answered, tr, te):.3f}")
print(f"random split,   colour it ignored           {probe(X, ignored, tr, te):.3f}     {probe(Xu, ignored, tr, te):.3f}")
print(f"held-out items, colour it answered with     {probe(X, answered, tr2, te2):.3f}     {probe(Xu, answered, tr2, te2):.3f}")
print(f"held-out items, colour it ignored           {probe(X, ignored, tr2, te2):.3f}     {probe(Xu, ignored, tr2, te2):.3f}")
print(f"shuffled labels on the trained model:       {probe(X, shuffled, tr, te):.3f}")
#                                             trained   untrained
# random split,   colour it answered with     1.000     0.566
# random split,   colour it ignored           0.598     0.572
# held-out items, colour it answered with     1.000     0.464
# held-out items, colour it ignored           0.513     0.224
# shuffled labels on the trained model:       0.234

Read the table row by row. The colour the model answers with probes at 1.000 against an untrained baseline of 0.566, which is a real and enormous effect, as it must be, since the model outputs it. The ignored colour probes at 0.598 against an untrained baseline of 0.572. The headline number was almost entirely random mixing, and the claim evaporates.

The held-out rows then flip the conclusion a second time. With items 36 to 47 excluded from the probe's training set, the ignored colour still probes at 0.513 while the untrained model drops to 0.224, below chance, because its memorised rules actively mislead on unseen items. So there is something real: training made colour linearly readable in a way that generalises to items the probe never saw. It is a different, smaller and better-supported claim than the one the first number suggested, and you only reach it by running both controls.

The capacity problem is the same story at a smaller scale. A linear probe on 32 dimensions fits 40 rows of pure noise perfectly, and the training accuracy tells you nothing.

from sklearn.linear_model import LogisticRegression as LR
for n_train in (40, 200, 2000):
    clf = LR(max_iter=5000, C=1e4).fit(X[:n_train], shuffled[:n_train])
    print(f"shuffled labels, {n_train:4d} training rows: "
          f"train {clf.score(X[:n_train], shuffled[:n_train]):.3f}  "
          f"test {clf.score(X[te], shuffled[te]):.3f}")
# shuffled labels,   40 training rows: train 1.000  test 0.236
# shuffled labels,  200 training rows: train 0.460  test 0.243
# shuffled labels, 2000 training rows: train 0.326  test 0.257

Even the surviving claim is about recoverability. Nothing above shows the model uses the ignored colour for anything, and in this task it provably does not, because changing item 2 while the cue says FIRST leaves the answer identical. Showing that a representation is used takes an intervention, which is the next method.

Activation patching#

This is the one that establishes causation. Run the model on two inputs, take the activation at one location from one run, insert it into the other, and see whether the output changes. If it does, that location carries information the output depends on.

clean run:      "The capital of France is" -> Paris
corrupted run:  "The capital of Italy is"  -> Rome

Patch one attention head's output from clean into corrupted.
If the answer flips to Paris, that head carries the country.

This is causal. Probing would only have shown the
information was present somewhere.

Three choices decide whether a patching result means anything, and all three are made before you run it. The corruption: a minimal edit that flips the answer, since a corruption that changes several things at once finds several things at once. The metric: the difference between the two answer logits, because accuracy is too coarse to register partial restoration and probability saturates near one. And the direction: patching clean activations into a corrupted run measures what is sufficient to restore the behaviour, while patching corrupted into clean measures what is necessary to destroy it, and the two can disagree when components back each other up. Say which one you ran.

Sparse autoencoders#

The response to superposition. Train an autoencoder with far more hidden units than the layer has dimensions, and a sparsity penalty so few activate at once. The hope is that it separates overlapping features into individually interpretable directions.

Two numbers describe any trained SAE and should be quoted together, because each alone can be made arbitrarily good at the other's expense. The reconstruction error says how much of the layer survives the round trip. The average number of features active per token, usually written L0, says how sparse the code is. A penalty weight of zero gives perfect reconstruction and a dense, uninterpretable code; a large penalty gives a beautifully sparse code that has thrown away the activation. Anyone reporting an SAE without both numbers has reported half of it.

The third number is the share of features that stop firing on anything and never recover, and it is routinely a third or more of the dictionary. Those are capacity you paid for and did not get.

It has produced genuinely interpretable features, and whether they are the model's features or artefacts of the autoencoder is an active question. Treat published feature interpretations as promising rather than settled.

What to carry into an interview#

• The four-row table above, as a sentence each. An interviewer asking about interpretability methods is usually testing whether you distinguish correlational from causal evidence, and that table is the answer.
• That a probe needs three controls named without prompting: an untrained model, shuffled labels, and a split that holds out entities. Volunteering them is the difference between a passable and a strong answer.
• That a patching result belongs to the prompt pair it was run on until replicated across many.
• Skip memorising SAE architecture variants. Knowing that reconstruction, L0 and the dead share must be reported together is worth more than knowing four names for the same encoder.

Interpretability work is mostly reading numbers out of a running model and checking whether they mean what you think. This section sets up the tooling for that: loading a small model with every internal value exposed, capturing them on a forward pass, and using three standard views, the residual stream, the logit lens and attention patterns, to watch one prediction form layer by layer. Everything later in this topic assumes you can get these values out and know which name refers to which point in the model.

Every activation, by name#

TransformerLens loads models such as GPT-2 small with a hook at every interesting point, so you can read or change any internal value. It is the standard tool for learning this field on small models.

The naming scheme is the part worth learning first, because every later technique is a matter of naming a location. A cache key is a component path and a hook point: blocks.5.hook_resid_post is the residual stream after block 5, blocks.5.attn.hook_z is the per-head output of block 5's attention before it is projected back, and the shorthand cache["pattern", 5] gives the attention weights. Shapes follow a convention too. Residual stream tensors are batch by position by model width; attention patterns are batch by head by query position by key position. Getting those two shapes into your fingers removes most of the confusion in the first week.

from transformer_lens import HookedTransformer

model = HookedTransformer.from_pretrained("gpt2-small")     # 12 layers, 12 heads
logits, cache = model.run_with_cache("The Eiffel Tower is in the city of")

print(model.to_string(logits[0, -1].argmax()))   # " Paris"
print(cache["blocks.5.hook_resid_post"].shape)   # [batch, position, d_model]
print(cache["pattern", 5].shape)                 # [batch, head, query_pos, key_pos]

The residual stream#

Think of a transformer as a stream of vectors, one per position, flowing from the embedding to the output. Each attention layer and each MLP reads from the stream and adds its result back. Nothing overwrites anything; everything is added. This view is what makes the rest of the field workable: you can ask what each component added.

Two consequences are worth stating explicitly. Because the final vector is a sum of contributions, the final logit for a token is a sum of each component's contribution to that logit, and you can attribute an output to components by decomposing that sum rather than by guessing. And because every component reads the whole stream, the stream works as a shared bus: a head in layer 2 can write something that only a head in layer 9 ever reads, with nothing in between touching it. That is why circuits span layers and why reading a single layer in isolation is rarely enough.

The logit lens#

Take the stream partway through the model and push it straight through the final layer norm and unembedding, as if the model stopped there. You get the model's best guess at each depth, and can watch the answer appear.

for layer in range(model.cfg.n_layers):
    resid = cache[f"blocks.{layer}.hook_resid_post"][:, -1:, :]
    guess = model.unembed(model.ln_final(resid))[0, -1].argmax()
    print(layer, repr(model.to_string(guess)))

The lens is useful and it rests on an assumption that is only roughly true: that an intermediate residual state lives in the same space as the final one, so the final unembedding is a fair way to read it. Early layers break that assumption badly, and a lens reading of layer 1 is close to meaningless. Treat the layer at which an answer first appears as a rough marker, not a measurement, and check anything you plan to claim from it by patching instead.

Attention patterns and induction heads#

An induction head completes repeated patterns: having seen 'Jack Kowalczyk' once, on seeing 'Jack' again it attends to the token after the earlier 'Jack' and predicts 'Kowalczyk'. It is one of the best understood circuits, and a good first target because you can find it yourself.

import torch

seq = torch.randint(1000, 10000, (1, 50))
tokens = torch.cat([seq, seq], dim=1)            # random tokens, repeated once
_, cache = model.run_with_cache(tokens)

for layer in range(model.cfg.n_layers):
    pattern = cache["pattern", layer][0]          # [head, q, k]
    # an induction head at position i attends to position i - 49 in the second half
    score = pattern.diagonal(offset=-49, dim1=-2, dim2=-1).mean(-1)
    for head in (score > 0.4).nonzero().flatten().tolist():
        print(f"layer {layer} head {head}: {score[head]:.2f}")

What an attention pattern does not tell you#

An attention pattern is the most seductive picture in the field. It is a heatmap, it is easy to produce, and it appears to show the model paying attention to the right words. What it actually shows is which positions a head reads from. It says nothing about what the head does with what it read, and nothing about whether removing the head would change the answer. Those are separate measurements, and the honest way to learn the difference is to make both on a model small enough to train in ten seconds.

The task below is three tokens: two items, then a cue saying which of the two to answer about. The answer is the colour of the cued item. Attention is written out by hand rather than taken from a library, because the point of the exercise is to switch one head off, and a head has to be visible before it can be removed.

import torch, torch.nn as nn

torch.manual_seed(0)
N_ITEMS, D, N_HEADS = 48, 32, 4
D_HEAD = D // N_HEADS
CUE_FIRST, CUE_SECOND = N_ITEMS, N_ITEMS + 1

def colour(item): return item % 4

class Block(nn.Module):
    """Attention written out so individual heads can be read and switched off."""
    def __init__(self):
        super().__init__()
        self.q, self.k, self.v, self.o = (nn.Linear(D, D) for _ in range(4))
        self.mlp = nn.Sequential(nn.Linear(D, 4 * D), nn.ReLU(), nn.Linear(4 * D, D))

    def forward(self, x, ablate=()):
        B, T, _ = x.shape
        q, k, v = (f(x).view(B, T, N_HEADS, D_HEAD).transpose(1, 2)
                   for f in (self.q, self.k, self.v))
        pattern = (q @ k.transpose(-1, -2) / D_HEAD ** 0.5).softmax(-1)   # [B,head,q,k]
        z = pattern @ v
        for h in ablate:
            z = z.clone(); z[:, h] = 0                    # switch one head off
        x = x + self.o(z.transpose(1, 2).reshape(B, T, D))
        return x + self.mlp(x), pattern

class Toy(nn.Module):
    def __init__(self):
        super().__init__()
        self.embed = nn.Embedding(N_ITEMS + 2, D)
        self.pos = nn.Parameter(torch.randn(3, D) * 0.02)
        self.blocks = nn.ModuleList([Block(), Block()])
        self.out = nn.Linear(D, 4)

    def forward(self, tokens, cache=None, ablate=()):
        x = self.embed(tokens) + self.pos
        for i, blk in enumerate(self.blocks):
            x, pattern = blk(x, ablate=[h for (layer, h) in ablate if layer == i])
            if cache is not None:
                cache[f"pattern_{i}"] = pattern.detach()
        return self.out(x[:, -1])

def make_batch(n):
    """[item 1, item 2, cue] -> the colour of whichever item the cue names."""
    a = torch.randint(0, N_ITEMS, (n,)); b = torch.randint(0, N_ITEMS, (n,))
    cue = torch.randint(0, 2, (n,))
    toks = torch.stack([a, b, torch.where(cue == 0, CUE_FIRST, CUE_SECOND)], 1)
    return toks, torch.where(cue == 0, colour(a), colour(b))

model = Toy()
opt = torch.optim.Adam(model.parameters(), lr=3e-3)
for step in range(800):
    toks, labels = make_batch(256)
    loss = nn.functional.cross_entropy(model(toks), labels)
    opt.zero_grad(); loss.backward(); opt.step()

Now measure both things for every head. How much of its attention, from the final position, goes to the item the cue selected; and what it costs to switch the head off. The cost needs two metrics, because one of them turns out to be useless.

toks, labels = make_batch(4000)
cued_first = toks[:, 2] == CUE_FIRST
cache = {}
with torch.no_grad(): model(toks, cache=cache)

def score(ablate=()):
    """Accuracy, and the margin between the right logit and the best wrong one."""
    with torch.no_grad(): lg = model(toks, ablate=ablate)
    right = lg.gather(1, labels[:, None]).squeeze(1)
    rest = lg.clone(); rest.scatter_(1, labels[:, None], -1e9)
    return (lg.argmax(-1) == labels).float().mean().item(), (right - rest.max(-1).values).mean().item()

base_acc, base_margin = score()
print(f"baseline: accuracy {base_acc:.3f}  margin {base_margin:.2f}\n")
print("head   attends to the cued item   accuracy   margin   margin lost")
attn, lost = [], []
for layer in range(2):
    pattern = cache[f"pattern_{layer}"][:, :, 2]           # queries from the last position
    for h in range(N_HEADS):
        w = torch.where(cued_first, pattern[:, h, 0], pattern[:, h, 1]).mean().item()
        a, m = score([(layer, h)])
        attn.append(w); lost.append(base_margin - m)
        print(f"L{layer}H{h}          {w:6.2f}             {a:.3f}   {m:6.2f}        {base_margin - m:5.2f}")
for name, group in (("all four heads in layer 0", [(0, h) for h in range(4)]),
                    ("all four heads in layer 1", [(1, h) for h in range(4)])):
    a, m = score(group)
    print(f"{name:26s}             {a:.3f}   {m:6.2f}        {base_margin - m:5.2f}")

baseline: accuracy 1.000  margin 18.41

head   attends to the cued item   accuracy   margin   margin lost
L0H0            0.67             1.000    16.05         2.36
L0H1            0.70             1.000    17.21         1.20
L0H2            0.56             1.000    17.39         1.02
L0H3            0.82             0.998    16.33         2.08
L1H0            0.93             0.993    14.10         4.31
L1H1            0.97             1.000    13.23         5.18
L1H2            0.59             1.000    16.64         1.77
L1H3            0.92             1.000    14.26         4.15
all four heads in layer 0              0.987    11.22         7.19
all four heads in layer 1              0.654     2.20        16.21

The accuracy column is the first lesson and it is a negative one. Seven of the eight heads can be deleted with accuracy still reading 1.000, and the eighth costs 0.007. If you had chosen accuracy as your metric you would have concluded that no head in this model matters, which is plainly false, because removing layer 1 entirely takes accuracy to 0.654. The margin column resolves every one of those ties. This is why interpretability work reports logit differences rather than accuracy, and it is worth having met the problem once.

The attention column is the second lesson. L1H1 attends 0.97 of its weight to the cued item and is also the most expensive head to remove, at 5.18, so here the picture and the intervention agree. But L1H2 attends only 0.59, less than L0H3's 0.82, and costs more than any head in layer 0. Across the eight heads the rank correlation between attention weight and margin lost is 0.88, which is high and is also the whole point: you do not know it is 0.88 rather than 0.1 until you run the ablation, and in a model with a genuine division of labour it is often far lower. Attention tells you where a head reads. Ablation tells you whether it matters. Publishing the first while describing the second is the commonest error in this area.

from scipy.stats import spearmanr
print(f"\nrank correlation, attention weight against margin lost: "
      f"{spearmanr(attn, lost).statistic:.2f}")
print(f"accuracy range across all eight single-head ablations: "
      f"{min(score([(l, h)])[0] for l in range(2) for h in range(4)):.3f} to 1.000")
# rank correlation, attention weight against margin lost: 0.88
# accuracy range across all eight single-head ablations: 0.993 to 1.000

One more thing the last two rows are telling you. The four layer-0 heads cost 1.02, 1.20, 2.08 and 2.36 individually, adding to 6.66, and 7.19 when removed together. Layer 1 adds to 15.41 individually and costs 16.21 together. Roughly additive in both cases, which is what you want to check rather than assume: when a group costs far less together than the sum of its parts, the components are redundant and are covering for each other, and single-head ablation will under-report all of them.

One caveat on how the ablation was done. Setting a head's output to zero moves the model off the distribution it was trained on, and some of the damage you measure is the model being confused by an impossible input rather than the head's job being missing. The stricter version replaces the head's output with its mean over a batch of inputs, which keeps the scale and removes only the input-dependent part. On this model the two give the same ordering; on larger models they sometimes do not, and a paper that does not say which it used has left out something load-bearing.

What to carry, and what to skip#

• Carry: residual stream shapes and attention pattern shapes. Nearly every bug in a first week of this work is a transposed axis.
• Carry: report logit differences, not accuracy. The table above is the reason, and it will come up in an interview as 'why not just measure accuracy'.
• Carry: an attention pattern is a hypothesis about a head, and an ablation is a test of it.
• Skip: hand-tuning heatmap plots. A printed table of eight numbers settles the question faster than a picture of it.
• Skip: trying to interpret attention in the first layer of a large model before you have replicated anything. Early layers are mostly doing token-level bookkeeping and are a bad place to start.

Reading activations tells you what is present in a model. Patching tells you what the output actually depends on, and it is the technique most interpretability claims rest on. This section runs one experiment end to end: two prompts differing by a single token, a metric that measures the decision you care about, and activations copied from one run into the other until you can see which locations carry the difference. It closes on what the result does and does not license you to say, which is where most of the mistakes are made.

The experiment#

Pick a behaviour. A well-studied one: in 'When Mary and John went to the store, John gave a drink to', GPT-2 predicts 'Mary', the name that was not repeated. Make a corrupted version that flips the answer by changing one token. Then copy activations from the clean run into the corrupted run, one location at a time, and see which ones restore the clean answer.

from functools import partial
import torch

clean   = model.to_tokens("When Mary and John went to the store, John gave a drink to")
corrupt = model.to_tokens("When Mary and John went to the store, Mary gave a drink to")
mary, john = model.to_single_token(" Mary"), model.to_single_token(" John")

def logit_diff(logits):
    return (logits[0, -1, mary] - logits[0, -1, john]).item()

_, clean_cache = model.run_with_cache(clean)
print(logit_diff(model(clean)), logit_diff(model(corrupt)))   # positive, then negative

Choosing the corruption is the decision that shapes everything afterwards, and it is made before any code runs. Swapping one name asks which components carry that name. Replacing the name with a random token asks a different question, and will light up components that handle unusual tokens rather than components that handle names. Replacing it with noise asks a third. None is wrong; they are three different experiments, and a result only means anything alongside a statement of which corruption produced it. The rule of thumb is that the corruption should change the answer and nothing else you can name.

Patch every layer and position#

def patch(resid, hook, pos):
    resid[:, pos, :] = clean_cache[hook.name][:, pos, :]
    return resid

n_pos = clean.shape[1]
result = torch.zeros(model.cfg.n_layers, n_pos)
for layer in range(model.cfg.n_layers):
    for pos in range(n_pos):
        logits = model.run_with_hooks(
            corrupt,
            fwd_hooks=[(f"blocks.{layer}.hook_resid_pre", partial(patch, pos=pos))],
        )
        result[layer, pos] = logit_diff(logits)
# plot result as a heatmap: layers down, positions across

Typically the information sits at the position of the changed name in early layers and moves to the final position in later layers. The heatmap shows when and where it travels. Patching the output of each attention head at the final position then narrows it down to the heads that move it.

The same experiment, small enough to run#

The three-token task below has a known answer, which is the point: you can check what patching says against what the task provably requires. Each input is an item, a second item, and a cue naming which of the two to answer about. The answer is the cued item's colour. The forward pass takes an optional patch argument that overwrites one position at one named location with a value copied from elsewhere, which is the whole of activation patching in four lines.

import torch, torch.nn as nn

torch.manual_seed(0)
N_ITEMS, D, N_HEADS = 48, 32, 4
CUE_FIRST, CUE_SECOND = N_ITEMS, N_ITEMS + 1

def colour(item): return item % 4          # the attribute the task uses
def shape(item):  return (item // 4) % 3   # in the token, never used by the task

class Block(nn.Module):
    def __init__(self):
        super().__init__()
        self.attn = nn.MultiheadAttention(D, N_HEADS, batch_first=True)
        self.mlp = nn.Sequential(nn.Linear(D, 4 * D), nn.ReLU(), nn.Linear(4 * D, D))

    def forward(self, x):
        x = x + self.attn(x, x, x, need_weights=False)[0]
        return x + self.mlp(x)

class Toy(nn.Module):
    def __init__(self):
        super().__init__()
        self.embed = nn.Embedding(N_ITEMS + 2, D)
        self.pos = nn.Parameter(torch.randn(3, D) * 0.02)
        self.blocks = nn.ModuleList([Block(), Block()])
        self.out = nn.Linear(D, 4)

    def forward(self, tokens, cache=None, patch=None):
        def at(name, x):
            if cache is not None:
                cache[name] = x.detach().clone()
            if patch is not None and patch[0] == name:
                x = x.clone(); x[:, patch[1]] = patch[2]     # copy one position in
            return x
        x = at("embeddings", self.embed(tokens) + self.pos)
        for i, blk in enumerate(self.blocks):
            x = at(f"after_block_{i}", blk(x))
        return self.out(x[:, -1])

def make_batch(n):
    a = torch.randint(0, N_ITEMS, (n,)); b = torch.randint(0, N_ITEMS, (n,))
    cue = torch.randint(0, 2, (n,))
    toks = torch.stack([a, b, torch.where(cue == 0, CUE_FIRST, CUE_SECOND)], 1)
    return toks, torch.where(cue == 0, colour(a), colour(b))

model = Toy()
opt = torch.optim.Adam(model.parameters(), lr=3e-3)
for step in range(800):
    toks, labels = make_batch(256)
    loss = nn.functional.cross_entropy(model(toks), labels)
    opt.zero_grad(); loss.backward(); opt.step()
with torch.no_grad():
    t, l = make_batch(4000)
    print(f"held-out accuracy: {(model(t).argmax(-1) == l).float().mean():.3f}")
# held-out accuracy: 1.000

Now the clean and corrupted pair. They differ in exactly one token, and that token flips the answer, which is what makes the difference between the two runs attributable to one thing. The metric is the difference between the two candidate logits.

# Item 4 is colour 0, item 5 is colour 1. One token differs between the two runs.
clean   = torch.tensor([[4, 9, CUE_FIRST]])          # the answer should be colour 0
corrupt = torch.tensor([[5, 9, CUE_FIRST]])          # the answer should be colour 1
ANSWER, ALTERNATIVE = colour(4), colour(5)

def logit_diff(logits):
    """Positive means the clean answer is winning. Exactly the decision we care about."""
    return (logits[0, ANSWER] - logits[0, ALTERNATIVE]).item()

clean_cache = {}
with torch.no_grad():
    hi = logit_diff(model(clean, cache=clean_cache))
    lo = logit_diff(model(corrupt))
print(f"clean run    {hi:+.2f}")
print(f"corrupted run {lo:+.2f}")
# clean run    +15.53
# corrupted run -28.12

Those two numbers are the ends of the scale. Patching a location restores some fraction of the distance from the corrupted value back to the clean one, and reporting that fraction rather than the raw logit makes locations comparable across prompt pairs with different absolute margins.

print("\nrecovered share of the clean logit difference, one location at a time")
print("                 pos 0 (item 1)  pos 1 (item 2)  pos 2 (cue)")
for name in ("embeddings", "after_block_0", "after_block_1"):
    row = []
    for pos in range(3):
        with torch.no_grad():
            out = model(corrupt, patch=(name, pos, clean_cache[name][:, pos]))
        row.append(100 * (logit_diff(out) - lo) / (hi - lo))
    print(f"{name:16s}" + "".join(f"{v:13.1f}%" for v in row))

recovered share of the clean logit difference, one location at a time
                 pos 0 (item 1)  pos 1 (item 2)  pos 2 (cue)
embeddings              100.0%          0.0%          0.0%
after_block_0            81.0%         -0.2%         21.4%
after_block_1             0.0%          0.0%        100.0%

Read the first column downwards. At the embeddings, patching position 0 restores everything, because at that depth position 0 is the only place the difference exists. After one block it restores 81% and the final position has picked up 21.4%: the copy is under way. After two blocks position 0 restores nothing, because by then the answer has been moved and position 0 is no longer read. The middle column is zero everywhere, correctly, since item 2 is not involved in this pair at all. That is what a clean patching result looks like, and having seen one makes a messy one on a real model much easier to interpret.

The control that makes it a result#

Patching found where the answer travels. It has not yet shown that the model is using the colour rather than the token. Each item also has a shape, which the task never asks about, and items 0, 4 and 8 all share colour 0 while differing in shape. If the model were keying on token identity rather than colour, swapping among those three would move the answer.

# The control: change the shape of item 1 and hold its colour fixed.
print("\nitems 0, 4 and 8 are all colour 0 and differ in shape")
for tok in (4, 0, 8, 5):
    with torch.no_grad():
        d = logit_diff(model(torch.tensor([[tok, 9, CUE_FIRST]])))
    print(f"  item {tok}: colour {colour(tok)}, shape {shape(tok)}  ->  logit diff {d:+7.2f}")
# items 0, 4 and 8 are all colour 0 and differ in shape
#   item 4: colour 0, shape 1  ->  logit diff  +15.53
#   item 0: colour 0, shape 0  ->  logit diff  +23.57
#   item 8: colour 0, shape 2  ->  logit diff  +13.13
#   item 5: colour 1, shape 1  ->  logit diff  -28.12

Changing the shape moves the margin around, from 13.13 to 23.57, and never changes the decision. Changing the colour takes it to -28.12 and flips it. So shape is present in the activations, since it visibly perturbs them, and the output does not depend on it. That pair of facts is the difference between probing and patching stated in four numbers, and it is worth quoting in an interview because it is concrete.

One pair is an anecdote#

Everything above came from a single prompt pair, which is exactly the mistake this field makes most often. The fix costs almost nothing on a model this size: run the same procedure over two hundred pairs with different items and report the spread rather than one number.

import statistics
torch.manual_seed(1)
rows = {name: [[] for _ in range(3)] for name in ("embeddings", "after_block_0", "after_block_1")}
for _ in range(200):
    a, b = torch.randint(0, N_ITEMS, (2,)).tolist()
    other = torch.randint(0, N_ITEMS, (1,)).item()
    if colour(a) == colour(other):
        continue                                  # the pair must flip the answer
    cl = torch.tensor([[a, b, CUE_FIRST]]); co = torch.tensor([[other, b, CUE_FIRST]])
    ans, alt = colour(a), colour(other)
    diff = lambda lg: (lg[0, ans] - lg[0, alt]).item()
    cc = {}
    with torch.no_grad():
        h = diff(model(cl, cache=cc)); l = diff(model(co))
    for name in rows:
        for pos in range(3):
            with torch.no_grad():
                out = model(co, patch=(name, pos, cc[name][:, pos]))
            rows[name][pos].append(100 * (diff(out) - l) / (h - l))

print("\nover 200 prompt pairs: median recovered share, and the 10th-90th percentile")
for name in rows:
    cells = []
    for pos in range(3):
        v = sorted(rows[name][pos]); n = len(v)
        cells.append(f"{statistics.median(v):5.0f}% [{v[n//10]:4.0f},{v[9*n//10]:4.0f}]")
    print(f"{name:16s}" + "  ".join(cells))
# over 200 prompt pairs: median recovered share, and the 10th-90th percentile
# embeddings        100% [ 100, 100]      0% [   0,   0]      0% [   0,   0]
# after_block_0      85% [  75,  95]      0% [  -0,   1]     14% [   5,  21]
# after_block_1       0% [   0,   0]      0% [   0,   0]    100% [ 100, 100]

The cost of that replication is worth noticing, because it is what decides whether you can afford it on a real model. Patching every location for one prompt pair is one forward pass per location: three depths by three positions is nine passes here, and twelve layers by fifteen positions is 180 on GPT-2 small. Two hundred pairs multiplies that by two hundred. On a toy model the whole thing finishes in five seconds; on a real model the same experiment is where an afternoon goes, which is the honest reason people report one pair. Batch the pairs rather than looping them and most of that cost disappears.

Reading it honestly#

• Patching shows a location is sufficient to carry the difference between these two prompts. Other prompts may use other routes.
• The choice of corruption decides what you find. A different corruption asks a different question.
• Components can compensate for each other, so knocking one out may show less effect than it really has.
• Restoring the clean answer is not the same as destroying it. Patching clean into corrupted measures sufficiency; patching corrupted into clean measures necessity. Say which you ran.
• A recovered share above 100% or below 0% is normal and means the patch overshot or moved things the wrong way. It is information, not a bug.

What to carry, and what to skip#

• Carry: clean prompt, minimally corrupted prompt, a logit difference, and a normalised recovered share. Those four decisions are the whole method and an interviewer will ask for them in that order.
• Carry: the shape control above. Being able to say in one sentence how you would show a representation is not used separates a strong answer from a recited one.
• Skip: patching every location before you have a hypothesis. Patch the residual stream by position first to find the route, then narrow to components along it.
• Skip: building a plotting pipeline. A printed table is enough to read a result and much faster to trust.

Individual neurons respond to several unrelated things at once, which is what makes them hard to read. A sparse autoencoder is the standard attempt to unpick that, and this section trains a small one on real activations: collecting the data, the two terms in the loss that pull against each other, the features that stop firing early in training and never recover, and how to write down what a feature means. That last part is the work, since a label taken from the twenty strongest examples is a hypothesis rather than a finding.

The idea again, briefly#

Neurons are hard to read because the model packs many features into overlapping directions. A sparse autoencoder learns a larger set of directions such that each activation is rebuilt from only a few of them. The hope is that each direction is one understandable feature.

A minimal version#

import torch, torch.nn as nn

class SAE(nn.Module):
    def __init__(self, d_model, n_features):
        super().__init__()
        self.enc = nn.Linear(d_model, n_features)
        self.dec = nn.Linear(n_features, d_model)

    def forward(self, x):
        f = torch.relu(self.enc(x))          # feature activations, mostly zero
        return self.dec(f), f

sae = SAE(768, 768 * 8)
opt = torch.optim.Adam(sae.parameters(), lr=1e-4)
for x in activation_batches:                  # e.g. layer 6 residual stream, many tokens
    x_hat, f = sae(x)
    loss = ((x_hat - x) ** 2).mean() + 1e-3 * f.abs().sum(-1).mean()
    opt.zero_grad(); loss.backward(); opt.step()

Two terms fight: reconstruct the activation well, and use as few features as possible. The penalty weight sets the balance. Track both the reconstruction error and the average number of active features per token.

Training one where the answer is known#

The difficulty with SAEs on a real model is that nobody knows what the right answer is, so every measurement of success is indirect. The way out for learning purposes is to build the activations yourself. The toy model below packs 64 sparse features into 16 dimensions, exactly the superposition situation an SAE is meant to undo, and because you built it you know every true feature direction. Train an SAE on its hidden activations and you can ask the question you can never ask on GPT-2: did it find them.

import torch

torch.manual_seed(0)
N_FEATURES, D_HIDDEN, SPARSITY = 64, 16, 0.95

class ToyModel(torch.nn.Module):
    """64 sparse features squeezed through 16 dimensions and reconstructed."""
    def __init__(self):
        super().__init__()
        self.W = torch.nn.Parameter(torch.randn(N_FEATURES, D_HIDDEN) * 0.1)
        self.b = torch.nn.Parameter(torch.zeros(N_FEATURES))

    def forward(self, x):
        return torch.relu((x @ self.W) @ self.W.T + self.b)

def batch(n):
    return torch.rand(n, N_FEATURES) * (torch.rand(n, N_FEATURES) < 1 - SPARSITY)

toy = ToyModel()
opt = torch.optim.Adam(toy.parameters(), lr=1e-2)
importance = 0.97 ** torch.arange(N_FEATURES)
for _ in range(3000):
    x = batch(1024)
    loss = (importance * (toy(x) - x) ** 2).mean()
    opt.zero_grad(); loss.backward(); opt.step()

W = toy.W.detach()
live = W.norm(dim=1) > 0.5
true = W[live] / W[live].norm(dim=1, keepdim=True)   # the ground-truth directions
print(f"{live.sum().item()} of {N_FEATURES} features represented in {D_HIDDEN} dimensions")
print(f"mean active features per sample: {(batch(4096) > 0).float().sum(-1).mean():.2f}")
# 57 of 64 features represented in 16 dimensions
# mean active features per sample: 3.25

Now train the autoencoder at four penalty weights and measure four things each time: how much of the activation survives the round trip, how many features fire per sample, how many never fire at all, and how close the learned directions come to the true ones. The last of those needs a control, which the next heading explains.

class SAE(torch.nn.Module):
    def __init__(self, d_in, n_feat):
        super().__init__()
        self.enc = torch.nn.Linear(d_in, n_feat)
        self.dec = torch.nn.Linear(n_feat, d_in, bias=False)

    def forward(self, h):
        f = torch.relu(self.enc(h))      # feature activations, mostly zero
        return self.dec(f), f

def train_sae(penalty, steps=3000):
    torch.manual_seed(1)
    sae = SAE(D_HIDDEN, 256)
    opt = torch.optim.Adam(sae.parameters(), lr=3e-3)
    for _ in range(steps):
        with torch.no_grad():
            h = batch(2048) @ W                        # the activations to explain
        h_hat, f = sae(h)
        recon = ((h_hat - h) ** 2).sum(-1).mean()
        loss = recon + penalty * f.abs().sum(-1).mean()
        opt.zero_grad(); loss.backward(); opt.step()
        with torch.no_grad():                          # unit-norm decoder columns
            sae.dec.weight.data /= sae.dec.weight.data.norm(dim=0, keepdim=True) + 1e-8
    return sae

def report(sae):
    with torch.no_grad():
        h = batch(16384) @ W
        h_hat, f = sae(h)
    fired = (f > 0).any(0)
    directions = sae.dec.weight.data.T[fired]
    best = (true @ directions.T).abs().max(-1).values      # recovery of each true feature
    torch.manual_seed(2)
    R = torch.randn(int(fired.sum()), D_HIDDEN); R = R / R.norm(dim=1, keepdim=True)
    control = (true @ R.T).abs().max(-1).values            # the same count, at random
    return (((h_hat - h) ** 2).sum(-1).mean().item(), (f > 0).float().sum(-1).mean().item(),
            int((~fired).sum()), best.mean().item(), control.mean().item())

print("penalty  reconstruction   L0   dead/256   recovery   random control")
for penalty in (0.01, 0.03, 0.10, 0.30):
    r, l0, dead, rec, ctrl = report(train_sae(penalty))
    print(f"  {penalty:4.2f}       {r:8.4f}  {l0:5.1f}      {dead:3d}       {rec:.2f}           {ctrl:.2f}")

penalty  reconstruction   L0   dead/256   recovery   random control
  0.01         0.0009   53.3       82       0.73           0.67
  0.03         0.0048   36.1      100       0.76           0.66
  0.10         0.0485   20.4      131       0.93           0.65
  0.30         0.1698    5.0      165       0.98           0.63

Read across the rows. Reconstruction error rises by a factor of 190 from the top row to the bottom, L0 falls from 53 to 5, the dead share doubles from 82 features to 165, and recovery of the true directions goes from 0.73 to 0.98. There is no setting that is good at everything, which is the actual content of the phrase 'the two terms fight'. The bottom row is the one that recovers the features, and it is also the one that throws away the most of the activation and wastes the most of the dictionary. Anyone showing you an SAE with one number has chosen which of these to tell you about.

The L0 column has a reference point here that a real model does not give you: the data genuinely has about 3.25 active features per sample, so the bottom row at 5.0 is roughly right and the top row at 53.3 is describing nothing. On a real layer you have no such anchor, which is why L0 is usually chosen by what stays interpretable rather than by what is correct.

The control that decides whether recovery means anything#

Recovery above is the best cosine similarity between each true direction and any of the SAE's directions. That statistic has a serious problem: with enough candidate directions in few enough dimensions, a random dictionary scores well. The control column is exactly that, the same number of directions drawn at random, and at a penalty of 0.01 the SAE's 0.73 against a random 0.67 is close to nothing. Only the bottom two rows clear their control by a margin worth reporting.

How large the random baseline is depends almost entirely on the width of the space, and it is worth knowing the shape of that curve before quoting any cosine similarity.

torch.manual_seed(2)
print("\nbest |cos| to the nearest of 100 random directions, by dimension")
for d in (5, 16, 64, 256):
    A = torch.randn(57, d); A = A / A.norm(dim=1, keepdim=True)
    B = torch.randn(100, d); B = B / B.norm(dim=1, keepdim=True)
    print(f"  {d:4d} dimensions: {(A @ B.T).abs().max(-1).values.mean():.2f}")
# best |cos| to the nearest of 100 random directions, by dimension
#      5 dimensions: 0.92
#     16 dimensions: 0.61
#     64 dimensions: 0.34
#    256 dimensions: 0.17

Dead features#

Many features stop activating on anything early in training and never recover. Check what share never fired over a large sample. Published SAE work uses various tricks, such as resampling dead features, to reduce this.

The dead column above is worth sitting with. At the setting that recovers the true features best, 165 of 256 dictionary entries never fire once in 16,384 samples. That is 64% of the capacity you trained, paid for and will store, doing nothing. It also quietly changes what the other numbers mean, because a dictionary of 256 that behaves as a dictionary of 91 is a different object from the one described in the configuration. Report the count of live features alongside the nominal size, always.

Interpreting a feature#

1. 1.Collect the text snippets where it activates most strongly and read them.
2. 2.Also read snippets where it activates weakly. A feature that looks like 'French' at the top may be 'any non-English text' lower down.
3. 3.Check causally: add the feature's direction to the stream and see whether output shifts the way the label predicts.

What to carry, and what to skip#

• Carry: reconstruction error, L0 and the live feature count are one measurement in three parts, and any of the three can be made to look good by sacrificing the others.
• Carry: what an SAE is for. It is an attempt to recover a sparse basis for a layer, not a decoder ring and not an explanation of a behaviour. A feature label is a hypothesis to be tested by intervention.
• Carry: any similarity number needs the random baseline for that dimensionality beside it.
• Skip: chasing the newest encoder variant. The measurement discipline above transfers to all of them; the architectures do not stay current.
• Skip: training an SAE on a real model as a first project. Train one where you know the answer, as above, then move up.

This section is about scoping a project rather than running a technique. It covers the order the work goes in, starting with reproducing a published result on a small model before asking anything of your own, then how narrow a behaviour has to be to be finishable, the controls that separate a real effect from a striking picture, and what a write-up of a small finding looks like. It is written against the usual failure: six months on a question too broad to answer, and a result nobody can check.

Replicate something first#

Before your own question, reproduce a published result on a small model: induction heads, or the name-mover heads in the indirect object identification work. It proves your tools work and teaches you what a real effect looks like, so you can tell one apart from noise later.

Choose small#

Good first projects study one narrow behaviour in a small model where you can check every step: a specific kind of fact recall, a simple pattern completion, a toy model trained on an algorithmic task. Broad questions such as 'how does the model reason' swallow months.

The test for whether a question is small enough is whether you can state, before starting, what the result table will look like: its rows, its columns, and which cell would change your mind. A question that cannot be turned into a table in ten minutes is not yet a project. 'Which heads move the subject to the final position in this sentence template' passes. 'How does the model do arithmetic' does not, and the version of it that does is 'does the model compute two-digit addition carry-by-carry, tested by patching the carry position across a hundred sums'.

Controls#

• Compare against random directions, random heads or shuffled labels. An effect that random components also produce is not an effect.
• Test across many prompts, not the one that worked.
• State in advance what result would contradict your hypothesis.

The third of those is the one people skip, and it is the one that stops the other two from being decoration. Interpretability work has an unusually large search space: several layers, dozens of heads, thousands of SAE features, and any number of properties you might probe for. If you look until something looks interesting, something will look interesting, and the number attached to it will be real. Here is that failure with nothing in the data at all.

import numpy as np
rng = np.random.default_rng(0)

# 300 activation vectors from a 64-dimensional layer, and a property of the input
# that the model does NOT encode: the labels are independent of the activations.
acts = rng.standard_normal((300, 64))
label = rng.integers(0, 2, 300)
train, test = slice(0, 150), slice(150, 300)

def accuracy(direction):
    proj = acts @ direction
    threshold = np.median(proj[train])
    pred = (proj[train] > threshold).astype(int)
    flip = (pred == label[train]).mean() < 0.5      # let the direction point either way
    p = ((acts[test] @ direction > threshold) ^ flip).astype(int)
    return (p == label[test]).mean()

preregistered = rng.standard_normal(64)
print(f"one direction chosen in advance:        {accuracy(preregistered):.3f}")

best = max(accuracy(rng.standard_normal(64)) for _ in range(500))
print(f"best of 500 directions searched over:   {best:.3f}")
print(f"the truth is that there is nothing here: 0.500")
# one direction chosen in advance:        0.533
# best of 500 directions searched over:   0.633
# the truth is that there is nothing here: 0.500

A 63% classifier for a property the model demonstrably does not encode, on a held-out split, is the shape of a great many interpretability results. The held-out split did not save it, because the search happened after the split and used the test set to choose. Nothing about the method was dishonest: someone looked at directions until one worked, which is what looking for a feature is.

The fix is to build the null distribution rather than reason about it. Run the same search procedure on data where you know there is nothing, and report where your finding sits in that distribution.

null = np.array([accuracy(rng.standard_normal(64)) for _ in range(500)])
print(f"null over random directions: median {np.median(null):.3f}  "
      f"95th percentile {np.quantile(null, 0.95):.3f}  max {null.max():.3f}")
# null over random directions: median 0.500  95th percentile 0.567  max 0.627

In practice that means writing down the search before running it: how many locations you will test, what threshold counts as a hit, and what you will do if nothing clears it. The last part is where the discipline lives, because the honest answer to a search that finds nothing is a short write-up saying so, and the tempting answer is one more search.

Write it up small#

A careful short write-up of a narrow finding, with code and the negative results, is valued in this field, and it is the most direct evidence you can offer that you can do the work.

A useful shape for that write-up, and for answering the interview question that asks you to describe a project: the behaviour in one sentence and why it is small enough to finish; the setup, meaning the clean and corrupted inputs and the metric; what you measured, with the spread across prompts rather than a single number; the controls you ran, named; and the thing you expected that did not happen. Five short sections. Work at this size is publishable as a blog post and is read, and it is the artefact that gets people hired into this area far more often than a long unfinished project on a frontier model.