Working out what the network is doing

We can tell that a model performs well, and not why. That gap matters whenever you need to trust it, debug it, or predict whether it will hold up on inputs unlike the ones you tested. Interpretability is the attempt to close it. This section covers the main approaches, from probing what information a representation carries to tracing the specific circuits that implement a behaviour, and is straight about what each can actually establish.

The problem#

We can measure that a model performs well and not why. That matters for trust, for debugging, for safety, and for knowing whether performance will hold on inputs unlike the test set.

What there is to look at#

Every technique here operates on activations, so it is worth being exact about what one is. A forward pass produces, at each layer and each token position, a vector of numbers as wide as the model. GPT-2 small is 12 layers and 768 dimensions, so a 20-token prompt leaves roughly 184,000 numbers behind at the layer boundaries alone, before counting anything inside attention or the MLPs. That pile is the whole subject matter. The job is to turn some of those numbers into a statement about the model that another person can check.

A feature is a direction, not a neuron#

The idea that makes the rest of the field workable is that the model's units of meaning are directions in activation space rather than individual coordinates. A direction is a unit vector as wide as the activation. The model writes a feature into the stream by adding a multiple of its direction, and reads it back by taking a dot product against a readout direction. Nothing more exotic is happening, and the whole mechanism fits in a few lines.

import torch

torch.manual_seed(0)
N_FEATURES, D_HIDDEN = 20, 5

W = torch.randn(N_FEATURES, D_HIDDEN)
W = W / W.norm(dim=1, keepdim=True)        # one unit direction per feature

x = torch.zeros(4, N_FEATURES)
x[0, 3] = 1.0                              # example 0: feature 3 on, alone
x[1, 7] = 1.0                              # example 1: feature 7 on, alone
x[2, 3] = 1.0; x[2, 7] = 0.5               # example 2: both, 7 at half strength
h = x @ W                                  # the activation: five numbers

print("activation for example 2:", [round(v, 2) for v in h[2].tolist()])
print("read feature 3 back:", [round(v, 2) for v in (h @ W[3]).tolist()])
print("read feature 7 back:", [round(v, 2) for v in (h @ W[7]).tolist()])
# activation for example 2: [0.03, -0.33, -0.65, 0.58, 0.57]
# read feature 3 back: [1.0, -0.06, 0.97, 0.0]
# read feature 7 back: [-0.06, 1.0, 0.44, 0.0]

Two consequences follow, and they set up everything else. A neuron is one coordinate, which is one particular direction out of infinitely many, and the model has no reason to prefer the coordinate axes; so asking what neuron 402 means is usually the wrong question. And because directions need not be orthogonal, features interfere, which is the -0.06 above. With few features relative to the width of the layer, that leakage is negligible. It is not negligible in a real model, and the reason is the next heading.

Superposition: more features than dimensions#

A central difficulty. Models represent more features than they have dimensions by packing them into overlapping directions, so a single neuron responds to several unrelated things. This is why reading individual neurons rarely works, and why sparse autoencoders became a major tool for separating features back out.

The reason the model gets away with it is sparsity. If most features are off in any given input, two features sharing an almost-identical direction rarely collide, and the interference costs less than not representing one of them at all. That is a claim you can test rather than accept, by squeezing twenty features through five dimensions at different sparsity levels and counting how many survive.

class ToyModel(torch.nn.Module):
    """Squeeze N features through D dimensions and try to get them all back."""
    def __init__(self):
        super().__init__()
        self.W = torch.nn.Parameter(torch.randn(N_FEATURES, D_HIDDEN) * 0.1)
        self.b = torch.nn.Parameter(torch.zeros(N_FEATURES))

    def forward(self, x):
        h = x @ self.W                                  # N -> D, the bottleneck
        return torch.relu(h @ self.W.T + self.b)        # D -> N, tied weights

def batch(sparsity, n=1024):
    magnitude = torch.rand(n, N_FEATURES)
    on = (torch.rand(n, N_FEATURES) < (1 - sparsity)).float()
    return magnitude * on

def train(sparsity, steps=4000):
    torch.manual_seed(0)
    m = ToyModel()
    opt = torch.optim.Adam(m.parameters(), lr=1e-2)
    importance = 0.9 ** torch.arange(N_FEATURES)        # earlier features matter more
    for _ in range(steps):
        x = batch(sparsity)
        loss = (importance * (m(x) - x) ** 2).mean()
        opt.zero_grad(); loss.backward(); opt.step()
    return m

for sparsity in (0.0, 0.7, 0.9, 0.99):
    m = train(sparsity)
    norms = m.W.norm(dim=1)
    live = norms > 0.5                                  # which features got a direction
    U = m.W / (norms[:, None] + 1e-8)
    cos = (U @ U.T).abs(); cos.fill_diagonal_(0)
    interference = cos[live][:, live].mean().item() if live.sum() > 1 else 0.0
    print(f"sparsity {sparsity:.2f}  represented {live.sum().item():2d}/20  "
          f"mean |cos| between them {interference:.2f}")
# sparsity 0.00  represented  5/20  mean |cos| between them 0.00
# sparsity 0.70  represented 10/20  mean |cos| between them 0.10
# sparsity 0.90  represented 12/20  mean |cos| between them 0.20
# sparsity 0.99  represented 18/20  mean |cos| between them 0.33

Read the two columns together. The dense row is the behaviour you would have predicted without ever hearing the word superposition: five dimensions, five features, no interference. Every row below it stores more features than the layer has dimensions, and the interference column is the price. A real layer sits at the bottom of that table, not the top, which is why a neuron in a trained network responds to a face, a piece of Python and the word Tuesday, and why nothing you can do to that neuron alone will separate them.

Approaches, roughly by depth#

• Attribution: which inputs mattered for this output. Useful and limited, since it says where the model looked and not what it concluded.
• Probing: train a small classifier on internal activations to test whether some property is represented there. Shows information is present, not that the model uses it.
• Activation patching: copy an activation from one run into another and see whether the output changes. The only one of the four that establishes a dependency rather than a correlation.
• Mechanistic interpretability: reverse-engineer the actual computation, identifying circuits that implement a behaviour. The most ambitious and the most informative when it works.

The useful way to hold these is by what a positive result entitles you to say. A probe that succeeds says the information is recoverable from that location by a linear map. A patch that changes the output says the model's computation at that location feeds the answer. Those are different claims, and the gap between them is where most disputed interpretability results live. Choose the method by the sentence you want to be able to write at the end, not by which one is easiest to run.

Honest about the state#

It is a real hiring area at the safety-focused labs, and one of the few where careful empirical work on small models still produces publishable results. It is also one of the few where an interviewer can tell within two questions whether you have run anything, because the answers that come from reading about the field and the answers that come from training a toy model do not sound alike.

What to carry, and what to skip#

• Carry: a feature is a direction, writing is addition along it, reading is a dot product. Every technique in the topic is a variation on those three sentences.
• Carry: superposition, and specifically why sparsity is what makes it pay. Without the sparsity half you cannot answer the follow-up.
• Carry: the distinction between what a probe establishes and what a patch establishes. It is the most frequently asked interpretability question outside a specialist team.
• Skip, for now: the taxonomy of named circuits in published papers. Knowing that induction heads exist is worth a sentence; reciting the list without having found one is worth nothing.
• Skip: choosing between saliency-map variants. The differences between them matter far less than the fact that none of them is causal.