Statistics for comparing training runs

You change something, retrain, and the number goes up by half a point. The only question that matters is whether you have found an improvement or looked at noise, and the honest answer is usually that you cannot tell yet. This section is about telling. There are two separate sources of randomness — which seed the run happened to get, and which examples happened to land in the test set — and they need different treatment, so start by knowing which one you are looking at.

Seed noise: why one run proves nothing#

Two training runs of the identical configuration, differing only in random seed, land on different numbers. Initialisation, data order and dropout masks all vary, and on a typical fine-tune the spread across seeds is around half a point to a point of accuracy — which is the same size as most claimed improvements. Here is a world where B really is better than A, by 0.4 points, and a single run of each is used to decide.

import numpy as np
rng = np.random.default_rng(0)

TRUE_A, TRUE_B, SEED_SD = 71.0, 71.4, 0.8   # B is genuinely better, by 0.4

def run(true, n=1):
    """One training run: the true quality plus seed noise."""
    return rng.normal(true, SEED_SD, n)

wins = sum(run(TRUE_B)[0] > run(TRUE_A)[0] for _ in range(10000))
print(f"single-seed comparison gets the direction right {wins/100:.0f}% of the time")
# single-seed comparison gets the direction right 63% of the time

The fix is more seeds, and the cost is set by the square root: averaging n runs divides the uncertainty in the average by the square root of n, so each halving of the error costs four times the compute. That is why the answer to how many seeds is small and specific rather than as many as possible.

seeds   standard error of the mean
  1            0.80 points
  3            0.46
  5            0.36
 10            0.25
 20            0.18

Test-set noise, and why pairing changes everything#

The second source of randomness is the test set itself: yours is one sample of the examples you could have collected, and a different sample would give a different number. A bootstrap estimates that by resampling your test set with replacement many times and watching how much the answer moves. The detail that matters more than the bootstrap itself is pairing — evaluating both models on the same resampled examples, rather than resampling each independently.

import numpy as np
rng = np.random.default_rng(0)

n_test = 2000
correct_a = rng.random(n_test) < 0.71          # per-example, not just the mean
correct_b = correct_a.copy()
correct_b[~correct_a & (rng.random(n_test) < 0.12)] = True    # B fixes some errors
correct_b[ correct_a & (rng.random(n_test) < 0.03)] = False   # and breaks a few

print(f"A {correct_a.mean():.3f}   B {correct_b.mean():.3f}")

def bootstrap_ci(paired):
    diffs = []
    for _ in range(10000):
        if paired:
            i = rng.integers(0, n_test, n_test)                  # same examples
            diffs.append(correct_b[i].mean() - correct_a[i].mean())
        else:
            ia = rng.integers(0, n_test, n_test)                 # different ones
            ib = rng.integers(0, n_test, n_test)
            diffs.append(correct_b[ib].mean() - correct_a[ia].mean())
    return np.percentile(diffs, [2.5, 97.5])

print("unpaired", np.round(bootstrap_ci(False), 3))
print("paired  ", np.round(bootstrap_ci(True), 3))

# A 0.708   B 0.715
# unpaired [-0.021  0.035]
# paired   [-0.003  0.017]

Now read what it says. The measured gap is 0.7 points, and the paired interval runs from minus 0.3 to plus 1.7. It contains zero, so this experiment cannot distinguish B from A, and reporting the 0.7 as an improvement would be reporting noise. That is the usual outcome of an honest first check, and saying so is the skill being examined.

Two ways forward, and neither is rerunning until it looks better. Get more test data, remembering that the interval narrows with the square root, so quadrupling the test set halves it — 2,000 examples gave a width of 0.021, 8,000 gives 0.010 and 32,000 gives 0.005. Or look directly at the disagreement, which is where all the information is.

b_only = int((~correct_a & correct_b).sum())   # B right, A wrong
a_only = int(( correct_a & ~correct_b).sum())   # A right, B wrong
print(b_only, a_only)
# 59 45

# Only the 104 examples where they disagree carry any signal; the 1,896
# they agree on cannot separate the models. 59 out of 104 is close enough
# to a coin flip that this test set cannot settle it. That is McNemar's
# test, and it is the right one for two models on the same examples.

Many comparisons#

Run twenty variants against a baseline at the usual five percent threshold and you should expect one to look significant with nothing behind it, because that is what five percent means. This is how a week of honest tinkering produces a result that does not survive. The defences are ordinary: decide what you are testing before you look, divide the threshold by the number of comparisons if you are going to make many, and treat anything found by a sweep as a hypothesis to confirm on fresh data rather than a finding.

Effect size#

Significance says an effect is probably not zero; it says nothing about whether the effect is worth having. With a large enough test set a 0.05-point gain becomes significant and remains useless. Report the difference in the units your reader cares about — points of accuracy, milliseconds, dollars per thousand requests — alongside its interval, and let the size and the certainty be read separately. A change that is significant and small should be described as significant and small.