Doing interpretability research

This section is about scoping a project rather than running a technique. It covers the order the work goes in, starting with reproducing a published result on a small model before asking anything of your own, then how narrow a behaviour has to be to be finishable, the controls that separate a real effect from a striking picture, and what a write-up of a small finding looks like. It is written against the usual failure: six months on a question too broad to answer, and a result nobody can check.

Replicate something first#

Before your own question, reproduce a published result on a small model: induction heads, or the name-mover heads in the indirect object identification work. It proves your tools work and teaches you what a real effect looks like, so you can tell one apart from noise later.

Choose small#

Good first projects study one narrow behaviour in a small model where you can check every step: a specific kind of fact recall, a simple pattern completion, a toy model trained on an algorithmic task. Broad questions such as 'how does the model reason' swallow months.

The test for whether a question is small enough is whether you can state, before starting, what the result table will look like: its rows, its columns, and which cell would change your mind. A question that cannot be turned into a table in ten minutes is not yet a project. 'Which heads move the subject to the final position in this sentence template' passes. 'How does the model do arithmetic' does not, and the version of it that does is 'does the model compute two-digit addition carry-by-carry, tested by patching the carry position across a hundred sums'.

Controls#

• Compare against random directions, random heads or shuffled labels. An effect that random components also produce is not an effect.
• Test across many prompts, not the one that worked.
• State in advance what result would contradict your hypothesis.

The third of those is the one people skip, and it is the one that stops the other two from being decoration. Interpretability work has an unusually large search space: several layers, dozens of heads, thousands of SAE features, and any number of properties you might probe for. If you look until something looks interesting, something will look interesting, and the number attached to it will be real. Here is that failure with nothing in the data at all.

import numpy as np
rng = np.random.default_rng(0)

# 300 activation vectors from a 64-dimensional layer, and a property of the input
# that the model does NOT encode: the labels are independent of the activations.
acts = rng.standard_normal((300, 64))
label = rng.integers(0, 2, 300)
train, test = slice(0, 150), slice(150, 300)

def accuracy(direction):
    proj = acts @ direction
    threshold = np.median(proj[train])
    pred = (proj[train] > threshold).astype(int)
    flip = (pred == label[train]).mean() < 0.5      # let the direction point either way
    p = ((acts[test] @ direction > threshold) ^ flip).astype(int)
    return (p == label[test]).mean()

preregistered = rng.standard_normal(64)
print(f"one direction chosen in advance:        {accuracy(preregistered):.3f}")

best = max(accuracy(rng.standard_normal(64)) for _ in range(500))
print(f"best of 500 directions searched over:   {best:.3f}")
print(f"the truth is that there is nothing here: 0.500")
# one direction chosen in advance:        0.533
# best of 500 directions searched over:   0.633
# the truth is that there is nothing here: 0.500

A 63% classifier for a property the model demonstrably does not encode, on a held-out split, is the shape of a great many interpretability results. The held-out split did not save it, because the search happened after the split and used the test set to choose. Nothing about the method was dishonest: someone looked at directions until one worked, which is what looking for a feature is.

The fix is to build the null distribution rather than reason about it. Run the same search procedure on data where you know there is nothing, and report where your finding sits in that distribution.

null = np.array([accuracy(rng.standard_normal(64)) for _ in range(500)])
print(f"null over random directions: median {np.median(null):.3f}  "
      f"95th percentile {np.quantile(null, 0.95):.3f}  max {null.max():.3f}")
# null over random directions: median 0.500  95th percentile 0.567  max 0.627

In practice that means writing down the search before running it: how many locations you will test, what threshold counts as a hit, and what you will do if nothing clears it. The last part is where the discipline lives, because the honest answer to a search that finds nothing is a short write-up saying so, and the tempting answer is one more search.

Write it up small#

A careful short write-up of a narrow finding, with code and the negative results, is valued in this field, and it is the most direct evidence you can offer that you can do the work.

A useful shape for that write-up, and for answering the interview question that asks you to describe a project: the behaviour in one sentence and why it is small enough to finish; the setup, meaning the clean and corrupted inputs and the metric; what you measured, with the spread across prompts rather than a single number; the controls you ran, named; and the thing you expected that did not happen. Five short sections. Work at this size is publishable as a blog post and is read, and it is the artefact that gets people hired into this area far more often than a long unfinished project on a frontier model.