Isolating what actually caused the improvement

An experiment that shows your change helped is not the same as an experiment that shows your change is what helped. Almost every result that fails to reproduce was produced by someone honest who changed two things, measured once, and credited the wrong one. This section is about the discipline that prevents it: a baseline given the same effort as your method, one variable moved at a time, an ablation for every component you are claiming credit for, and enough seeds that the number means something.

A fair baseline#

The baseline is the part everyone under-resources, because there is no reward for making the thing you are trying to beat look good. A comparison is only informative if the baseline got what your method got: the same tuning budget, the same number of training steps, the same data cleaning, the same amount of your attention. A tuned method against an out-of-the-box baseline measures your tuning.

• Was the baseline tuned over the same number of configurations as the method?
• Did both get the same training budget — steps, epochs, wall-clock, or tokens?
• Did the baseline get the data cleaning, deduplication or filtering the method got?
• Is the baseline the strongest simple thing, or the first thing that came to hand?
• Would a specialist in the baseline recognise it as a fair attempt?

The check is cheap and it has two possible outcomes, both useful. Comparing gradient boosting against a default logistic regression on one tabular dataset showed a gain of 0.082; giving the baseline scaling and a small sweep over its regularisation moved it from 0.835 to 0.838, leaving 0.078. The gain survived, and it is now defensible in a way it was not before the check.

One variable at a time#

The other half is moving one thing per experiment. It feels slow, and it is the only way to know which change did the work. Here two changes are made together — ten times as many boosting iterations and half the learning rate — and the pair produces a modest gain.

from sklearn.datasets import make_classification
from sklearn.ensemble import HistGradientBoostingClassifier as GBM
from sklearn.model_selection import cross_val_score

X, y = make_classification(n_samples=4000, n_features=25, n_informative=8,
                           class_sep=0.7, random_state=0)

def score(**kw):
    return cross_val_score(GBM(random_state=0, **kw), X, y, cv=5).mean()

baseline = score(max_iter=40,  learning_rate=0.10)
new      = score(max_iter=400, learning_rate=0.05)   # two changes at once
print(round(baseline, 4), round(new, 4), round(new - baseline, 4))
# 0.9098 0.917 0.0072   "the new configuration is 0.7 points better"

A paper could stop here and report a 0.7 point improvement from the new configuration. Now move one variable at a time instead.

print(round(score(max_iter=400, learning_rate=0.10), 4))   # iterations only
print(round(score(max_iter=40,  learning_rate=0.05), 4))   # learning rate only
# 0.9198   +0.0100 against the baseline
# 0.8953   -0.0145 against the baseline

So the honest report is not that the new configuration helps. It is that training longer helps, that halving the learning rate hurts, and that doing both is worse than simply training longer. Anyone who adopted the pair as a unit would inherit a change that costs them accuracy, and would have no way to discover it, because the combined number looked fine.

Ablations#

An ablation removes one component and re-measures. The rule is simple: if you claim credit for a component, you owe an ablation for it, and a component whose removal costs nothing should be removed from the method rather than defended in the paper. Build the table as you go, because reconstructing it at the end is how experiments get skipped.

configuration                     accuracy   delta vs full
full method                         0.9198          —
  without the auxiliary loss        0.9191       -0.0007   drop the component
  without the curriculum            0.9102       -0.0096   real, and the largest
  without the new tokenizer         0.9163       -0.0035   real but small
  without curriculum AND aux loss   0.9088       -0.0110   roughly additive

Two components can also be redundant, where removing either alone costs nothing because the other covers for it, and removing both is expensive. An ablation that only ever removes one thing at a time will report that neither matters, and the method will then be simplified into something that does not work.

Seeds and variance#

Every number above is one run. Seed noise on a typical training setup is around half a point to a point, which is larger than several of the deltas in that ablation table, so each row needs at least three seeds before its sign can be trusted. The arithmetic for how many, and how to compare two runs once you have them, is in the statistics section of this topic. The practical rule: if a delta is smaller than your seed spread, you have not measured it yet, and reporting it as a finding is the single fastest way to publish something that will not replicate.