Experiment design

An experiment that shows your change helped is not the same as an experiment that shows your change is what helped. Almost every result that fails to reproduce was produced by someone honest who changed two things, measured once, and credited the wrong one. This section is about the discipline that prevents it: a baseline given the same effort as your method, one variable moved at a time, an ablation for every component you are claiming credit for, and enough seeds that the number means something.

A fair baseline#

The baseline is the part everyone under-resources, because there is no reward for making the thing you are trying to beat look good. A comparison is only informative if the baseline got what your method got: the same tuning budget, the same number of training steps, the same data cleaning, the same amount of your attention. A tuned method against an out-of-the-box baseline measures your tuning.

• Was the baseline tuned over the same number of configurations as the method?
• Did both get the same training budget — steps, epochs, wall-clock, or tokens?
• Did the baseline get the data cleaning, deduplication or filtering the method got?
• Is the baseline the strongest simple thing, or the first thing that came to hand?
• Would a specialist in the baseline recognise it as a fair attempt?

The check is cheap and it has two possible outcomes, both useful. Comparing gradient boosting against a default logistic regression on one tabular dataset showed a gain of 0.082; giving the baseline scaling and a small sweep over its regularisation moved it from 0.835 to 0.838, leaving 0.078. The gain survived, and it is now defensible in a way it was not before the check.

One variable at a time#

The other half is moving one thing per experiment. It feels slow, and it is the only way to know which change did the work. Here two changes are made together — ten times as many boosting iterations and half the learning rate — and the pair produces a modest gain.

from sklearn.datasets import make_classification
from sklearn.ensemble import HistGradientBoostingClassifier as GBM
from sklearn.model_selection import cross_val_score

X, y = make_classification(n_samples=4000, n_features=25, n_informative=8,
                           class_sep=0.7, random_state=0)

def score(**kw):
    return cross_val_score(GBM(random_state=0, **kw), X, y, cv=5).mean()

baseline = score(max_iter=40,  learning_rate=0.10)
new      = score(max_iter=400, learning_rate=0.05)   # two changes at once
print(round(baseline, 4), round(new, 4), round(new - baseline, 4))
# 0.9098 0.917 0.0072   "the new configuration is 0.7 points better"

A paper could stop here and report a 0.7 point improvement from the new configuration. Now move one variable at a time instead.

print(round(score(max_iter=400, learning_rate=0.10), 4))   # iterations only
print(round(score(max_iter=40,  learning_rate=0.05), 4))   # learning rate only
# 0.9198   +0.0100 against the baseline
# 0.8953   -0.0145 against the baseline

So the honest report is not that the new configuration helps. It is that training longer helps, that halving the learning rate hurts, and that doing both is worse than simply training longer. Anyone who adopted the pair as a unit would inherit a change that costs them accuracy, and would have no way to discover it, because the combined number looked fine.

Ablations#

An ablation removes one component and re-measures. The rule is simple: if you claim credit for a component, you owe an ablation for it, and a component whose removal costs nothing should be removed from the method rather than defended in the paper. Build the table as you go, because reconstructing it at the end is how experiments get skipped.

configuration                     accuracy   delta vs full
full method                         0.9198          —
  without the auxiliary loss        0.9191       -0.0007   drop the component
  without the curriculum            0.9102       -0.0096   real, and the largest
  without the new tokenizer         0.9163       -0.0035   real but small
  without curriculum AND aux loss   0.9088       -0.0110   roughly additive

Two components can also be redundant, where removing either alone costs nothing because the other covers for it, and removing both is expensive. An ablation that only ever removes one thing at a time will report that neither matters, and the method will then be simplified into something that does not work.

Seeds and variance#

Every number above is one run. Seed noise on a typical training setup is around half a point to a point, which is larger than several of the deltas in that ablation table, so each row needs at least three seeds before its sign can be trusted. The arithmetic for how many, and how to compare two runs once you have them, is in the statistics section of this topic. The practical rule: if a delta is smaller than your seed spread, you have not measured it yet, and reporting it as a finding is the single fastest way to publish something that will not replicate.

A confound is anything that changed alongside your variable and could explain the result instead of it. They are not exotic — the usual ones are that the two arms saw different data, ran for different lengths of time, or were evaluated on different examples — and they are dangerous because the experiment still produces a clean number. This section is about the habits that catch them: matching the control to the treatment, deciding the analysis before seeing the data, and publishing the runs that did not work.

A confound that survives every check you would think to run#

Two prompting strategies are compared on a benchmark that mixes easy and hard questions. Strategy B was the new idea, so it was tried mostly on the hard questions; A, already trusted, was used mostly on the easy ones. Nobody did anything dishonest, and every individual number is correct.

rows = [                     # (subset, strategy, asked, correct)
    ("easy", "A", 1000, 950),
    ("easy", "B",  100,  97),
    ("hard", "A",  100,  40),
    ("hard", "B", 1000, 450),
]

totals = {}
for subset, strategy, n, c in rows:
    a, b = totals.get(strategy, (0, 0))
    totals[strategy] = (a + n, b + c)
    print(f"{subset:5} {strategy}  {c}/{n} = {c/n:.3f}")

for strategy, (n, c) in totals.items():
    print(f"overall {strategy}: {c}/{n} = {c/n:.3f}")

# easy  A  950/1000 = 0.950
# easy  B    97/100 = 0.970
# hard  A    40/100 = 0.400
# hard  B  450/1000 = 0.450
# overall A: 990/1100 = 0.900
# overall B: 547/1100 = 0.497

The overall number is measuring question difficulty, not strategy, because difficulty was not held constant between the arms. This is Simpson's paradox, and the reason to know it by name is that the reversal can go either way: the same imbalance can manufacture an improvement just as easily as it destroyed one here. Any aggregate over a mixture can be reversed by changing the mixture.

Controls that match the treatment#

A control arm is only a control if it differs from the treatment in exactly one respect. In practice several things move together, because the new method arrived with a new codebase, a new data loader and a different default. The discipline is to write down what differs before running anything, and to make the list as short as one line.

what must match                        why it is usually violated
the evaluation set, exactly            the new arm was tested on newer data
the training budget                    the new method "needed" longer to converge
the tuning effort                      the method got a sweep, the baseline defaults
the preprocessing                      the new pipeline also fixed a data bug
the random seeds                       one arm got lucky, and nobody checked
the metric and its averaging           macro in one table, micro in another
the person and the week                enthusiasm and skill are not constant

Pre-registering your analysis#

Pre-registration sounds like bureaucracy and is really a defence against your own cleverness. Before the runs finish, write down the metric, the subgroups you will report, how many seeds, and what result would count as the method failing. It takes five minutes, and it converts the most common failure — trying analyses until one is favourable — into something you would notice yourself doing.

• The primary metric, chosen now, and the threshold that counts as success.
• The subgroups you will break the result down by, listed in advance so they cannot be chosen afterwards.
• Number of seeds, and the spread you expect from them.
• What you will do if the result is ambiguous, which is the most likely outcome.
• What result would make you abandon the method. If there is none, you are not running an experiment.

It does not forbid looking further. Exploratory analysis is how findings are made, and the rule is only that it is labelled: this was the planned comparison, and these are things we noticed afterwards and would need to confirm on fresh data. Reviewers and interviewers both accept exploratory results readily when they are named as such, and distrust an entire paper when they suspect a planned result was chosen after the fact.

Negative results#

Most things you try will not work, and the record of what failed is worth more than it is given credit for — to you in six months, to the next person, and in an interview, where 'what have you tried that did not work' is a standard question and a vague answer is a poor signal. Keep the failed runs with enough detail to be interpretable: what you tried, what you expected, what happened, and your best guess at why.

The distinction that makes a negative result useful is between a method that does not work and an experiment that could not have detected it working. A run where the deltas were smaller than the seed spread has not shown the method fails; it has shown the experiment was underpowered, which is a different sentence and a much more honest one. Recording which of the two you have is most of the value.

You change something, retrain, and the number goes up by half a point. The only question that matters is whether you have found an improvement or looked at noise, and the honest answer is usually that you cannot tell yet. This section is about telling. There are two separate sources of randomness — which seed the run happened to get, and which examples happened to land in the test set — and they need different treatment, so start by knowing which one you are looking at.

Seed noise: why one run proves nothing#

Two training runs of the identical configuration, differing only in random seed, land on different numbers. Initialisation, data order and dropout masks all vary, and on a typical fine-tune the spread across seeds is around half a point to a point of accuracy — which is the same size as most claimed improvements. Here is a world where B really is better than A, by 0.4 points, and a single run of each is used to decide.

import numpy as np
rng = np.random.default_rng(0)

TRUE_A, TRUE_B, SEED_SD = 71.0, 71.4, 0.8   # B is genuinely better, by 0.4

def run(true, n=1):
    """One training run: the true quality plus seed noise."""
    return rng.normal(true, SEED_SD, n)

wins = sum(run(TRUE_B)[0] > run(TRUE_A)[0] for _ in range(10000))
print(f"single-seed comparison gets the direction right {wins/100:.0f}% of the time")
# single-seed comparison gets the direction right 63% of the time

The fix is more seeds, and the cost is set by the square root: averaging n runs divides the uncertainty in the average by the square root of n, so each halving of the error costs four times the compute. That is why the answer to how many seeds is small and specific rather than as many as possible.

seeds   standard error of the mean
  1            0.80 points
  3            0.46
  5            0.36
 10            0.25
 20            0.18

Test-set noise, and why pairing changes everything#

The second source of randomness is the test set itself: yours is one sample of the examples you could have collected, and a different sample would give a different number. A bootstrap estimates that by resampling your test set with replacement many times and watching how much the answer moves. The detail that matters more than the bootstrap itself is pairing — evaluating both models on the same resampled examples, rather than resampling each independently.

import numpy as np
rng = np.random.default_rng(0)

n_test = 2000
correct_a = rng.random(n_test) < 0.71          # per-example, not just the mean
correct_b = correct_a.copy()
correct_b[~correct_a & (rng.random(n_test) < 0.12)] = True    # B fixes some errors
correct_b[ correct_a & (rng.random(n_test) < 0.03)] = False   # and breaks a few

print(f"A {correct_a.mean():.3f}   B {correct_b.mean():.3f}")

def bootstrap_ci(paired):
    diffs = []
    for _ in range(10000):
        if paired:
            i = rng.integers(0, n_test, n_test)                  # same examples
            diffs.append(correct_b[i].mean() - correct_a[i].mean())
        else:
            ia = rng.integers(0, n_test, n_test)                 # different ones
            ib = rng.integers(0, n_test, n_test)
            diffs.append(correct_b[ib].mean() - correct_a[ia].mean())
    return np.percentile(diffs, [2.5, 97.5])

print("unpaired", np.round(bootstrap_ci(False), 3))
print("paired  ", np.round(bootstrap_ci(True), 3))

# A 0.708   B 0.715
# unpaired [-0.021  0.035]
# paired   [-0.003  0.017]

Now read what it says. The measured gap is 0.7 points, and the paired interval runs from minus 0.3 to plus 1.7. It contains zero, so this experiment cannot distinguish B from A, and reporting the 0.7 as an improvement would be reporting noise. That is the usual outcome of an honest first check, and saying so is the skill being examined.

Two ways forward, and neither is rerunning until it looks better. Get more test data, remembering that the interval narrows with the square root, so quadrupling the test set halves it — 2,000 examples gave a width of 0.021, 8,000 gives 0.010 and 32,000 gives 0.005. Or look directly at the disagreement, which is where all the information is.

b_only = int((~correct_a & correct_b).sum())   # B right, A wrong
a_only = int(( correct_a & ~correct_b).sum())   # A right, B wrong
print(b_only, a_only)
# 59 45

# Only the 104 examples where they disagree carry any signal; the 1,896
# they agree on cannot separate the models. 59 out of 104 is close enough
# to a coin flip that this test set cannot settle it. That is McNemar's
# test, and it is the right one for two models on the same examples.

Many comparisons#

Run twenty variants against a baseline at the usual five percent threshold and you should expect one to look significant with nothing behind it, because that is what five percent means. This is how a week of honest tinkering produces a result that does not survive. The defences are ordinary: decide what you are testing before you look, divide the threshold by the number of comparisons if you are going to make many, and treat anything found by a sweep as a hypothesis to confirm on fresh data rather than a finding.

Effect size#

Significance says an effect is probably not zero; it says nothing about whether the effect is worth having. With a large enough test set a 0.05-point gain becomes significant and remains useless. Report the difference in the units your reader cares about — points of accuracy, milliseconds, dollars per thousand requests — alongside its interval, and let the size and the certainty be read separately. A change that is significant and small should be described as significant and small.

An ablation study answers one question for each part of your method: if this were removed, would the result change? It is the difference between a method and a pile of things that happened to be in the code when the good number appeared. This section runs a real one — five components of an ordinary text classifier, each removed in turn — and the outcome is the usual one, which is that most of them earn nothing and one is actively harmful.

Remove one thing at a time#

The method here is a TF-IDF text classifier over four newsgroups, with five deliberate choices: lowercasing, stopword removal, bigrams, sublinear term frequency, and dropping terms that appear in only one document. Each is the kind of thing that ends up in a method section as though it were load-bearing.

from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline

cats = ["sci.med", "sci.space", "rec.autos", "talk.politics.guns"]
data = fetch_20newsgroups(subset="train", categories=cats,
                          remove=("headers", "footers", "quotes"), random_state=0)

def score(**override):
    opts = dict(lowercase=True, stop_words="english", ngram_range=(1, 2),
                sublinear_tf=True, min_df=2)
    opts.update(override)                       # one change per call
    pipe = make_pipeline(TfidfVectorizer(**opts), LogisticRegression(max_iter=2000))
    return cross_val_score(pipe, data.data, data.target, cv=5).mean()

full = score()
print(f"full                   {full:.4f}")
for label, change in [("no bigrams",          dict(ngram_range=(1, 1))),
                      ("no stopword removal", dict(stop_words=None)),
                      ("no lowercasing",      dict(lowercase=False)),
                      ("no sublinear tf",     dict(sublinear_tf=False)),
                      ("no min_df filter",    dict(min_df=1))]:
    s = score(**change)
    print(f"{label:22} {s:.4f}   {s - full:+.4f}")

# full                   0.8883
# no bigrams             0.8861   -0.0022
# no stopword removal    0.8754   -0.0129
# no lowercasing         0.8840   -0.0043
# no sublinear tf        0.8887   +0.0004
# no min_df filter       0.8956   +0.0073

This is the ordinary result of an honest ablation, and it is why the exercise is worth doing on your own work before a reviewer does it on your behalf. Of five components, one matters (stopword removal, 1.3 points), two are marginal, one is inert, and one is harmful. The method should now lose two of its five parts, which makes it simpler to describe, faster to run and easier for someone else to reproduce.

Interactions: why one at a time is not enough#

Removing one thing at a time is necessary and not sufficient, because components can cover for each other. Take the two most interesting rows above and remove both together. Individually they sum to a loss of 0.56 points. Together they lose 1.50.

both = score(stop_words=None, min_df=1)
singles = (score(stop_words=None) - full) + (score(min_df=1) - full)
print(f"both removed  {both:.4f}   {both - full:+.4f}")
print(f"sum of singles                {singles:+.4f}")

# both removed  0.8732   -0.0150
# sum of singles                -0.0056

So the honest conclusion is not the one the single-ablation table suggested. The min_df filter is not useless — it is redundant with stopword removal on this data, and which of the two you keep is a choice rather than a finding. A table of single ablations alone would have reported the wrong thing confidently, which is why the pairs you suspect of interacting are worth the extra runs.

Budgeting the compute#

Every pair is not affordable: n components give n single ablations, and n times n minus one over two pairs — five components is five and ten, twenty-five components is twenty-five and three hundred. So spend the budget deliberately rather than exhaustively.

• Every component you claim credit for gets a single ablation. This is not optional; it is the table.
• Pairs only where you have a reason to suspect interaction — two things addressing the same problem, as above.
• Ablate at the smallest scale where the effect is still visible, then confirm the two or three that matter at full scale.
• Run the ablations as you build, not at the end. A component ablated the week it was added costs one run; ablated at the end it costs a rerun of everything.
• Seeds before extra ablations: three seeds on five components beats one seed on fifteen, because a single-seed delta smaller than the seed spread is not a measurement.

What reviewers will ask#

question                                  what it is really checking
"Did you ablate X?"                       whether X earns its place
"What is the variance on these deltas?"   whether the deltas exist at all
"Does the ordering hold on dataset B?"    whether this is a method or a dataset
"What happens if you remove two?"         whether the components are redundant
"Why is component Y in the method         whether you read your own table
  if removing it costs 0.0004?"
"What would have made you drop this?"     whether the study could have failed