Reproducing a paper

Reproducing a result teaches more per hour than reading does, because it forces every detail the paper left implicit out into the open. This section covers choosing a paper you can afford to reproduce, working from the paper before the code, starting small enough that the first run finishes, diagnosing a mismatch in the order that finds it fastest, and writing up what you found. It starts with a reproduction you can run in the next two minutes.

One you can run right now#

Double descent is a well-known result: as a model grows past the point where it can fit the training data exactly, test error rises to a peak and then falls again, ending up better than any of the smaller models. It contradicts the bias-variance picture most courses teach, it was the subject of a widely cited paper, and it reproduces on a laptop in about fifteen lines. Forty training examples, and a random-feature model whose width is swept from far below that to ten times it.

import numpy as np
rng = np.random.default_rng(0)

n_train, n_test, d = 40, 500, 20
w = rng.normal(size=d)

def make(n):
    X = rng.normal(size=(n, d))
    return X, X @ w + rng.normal(0, 0.5, n)

Xtr, ytr = make(n_train)
Xte, yte = make(n_test)

P = rng.normal(size=(d, 400))            # a pool of random projections
for p in (5, 20, 38, 40, 42, 50, 80, 200, 400):
    Ftr = np.tanh(Xtr @ P[:, :p])        # p random features
    Fte = np.tanh(Xte @ P[:, :p])
    beta, *_ = np.linalg.lstsq(Ftr, ytr, rcond=None)   # minimum-norm fit
    print(f"p={p:4d}  test MSE {np.mean((Fte @ beta - yte) ** 2):10.3f}")

# p=   5  test MSE     12.834
# p=  20  test MSE     13.360
# p=  38  test MSE    228.365
# p=  40  test MSE    248.771      <- p = n, the interpolation threshold
# p=  42  test MSE    183.613
# p=  50  test MSE     36.164
# p=  80  test MSE      5.108
# p= 200  test MSE      3.693
# p= 400  test MSE      3.106      <- ten times as many parameters as data

That is what a reproduction feels like when it works, and it is why the exercise is worth the hours. You now know something is true rather than having read that it is, you have a knob you can turn, and the next questions arrive on their own — why does the peak sit there, what happens with regularisation added, what happens with more training data. Each is a short edit away.

Choose well#

Most reproductions fail at the choosing step, because the paper needed a cluster and a month. Pick for feasibility first and interest second, since an interesting paper you cannot finish teaches nothing.

• The compute fits what you have, or the result has a small-scale version that keeps the phenomenon.
• The data is public and downloadable today, not on request.
• The claim is specific enough to be checked — a number in a table, not a general improvement.
• There is official code, ideally one you deliberately do not read until your version runs.
• The result is surprising to you. Reproducing something you already believe teaches less.

Implement from the paper, then compare#

Write your version from the paper first. Only then read the official code, and list every difference. Official code routinely contains details the paper omits — a learning rate warmup, a normalisation, a data filter, a gradient clip, an evaluation that drops the last incomplete batch. That list of differences is the most valuable artefact the exercise produces, and it is the thing you can only get by implementing first. Read the code first and you will simply absorb the details without ever noticing they were missing.

Start small#

Get the method working on a tiny version of the problem first: fewer layers, less data, shorter training. Confirm your baseline matches the paper's baseline at that scale before you test the method at all. If the baseline is off, the method comparison is meaningless and every hour spent on it is wasted. A run you can complete in two minutes is worth more than a faithful one you can run twice.

When numbers do not match#

They usually will not, and the causes have a reliable order. Work down this list rather than rereading the method, because the method is rarely the problem.

check first, it is usually here
  evaluation mismatch   different split, different metric, different averaging
  preprocessing         tokenizer, normalisation, resizing, deduplication
  seed variance         is your gap smaller than the run-to-run spread?
  training budget       steps, epochs, batch size, effective batch after accumulation
  schedule details      warmup, decay shape, gradient clipping
  numerical precision   fp16 vs bf16 vs fp32, and where the loss is accumulated

check last, it is rarely here
  the method itself     the equation you implemented from the paper

Write it up#

A short report saying what you reproduced, what differed, which undocumented details you found, and a link to your code is a genuinely useful public artefact — and for research roles it is some of the most persuasive evidence you can offer, because it is the only kind that shows what you do when a number does not come out right. Keep it to a page: the claim, your setup, a table of your numbers against the paper's, the differences you found, and what you would check next.