Controls, confounds and pre-registration

A confound is anything that changed alongside your variable and could explain the result instead of it. They are not exotic — the usual ones are that the two arms saw different data, ran for different lengths of time, or were evaluated on different examples — and they are dangerous because the experiment still produces a clean number. This section is about the habits that catch them: matching the control to the treatment, deciding the analysis before seeing the data, and publishing the runs that did not work.

A confound that survives every check you would think to run#

Two prompting strategies are compared on a benchmark that mixes easy and hard questions. Strategy B was the new idea, so it was tried mostly on the hard questions; A, already trusted, was used mostly on the easy ones. Nobody did anything dishonest, and every individual number is correct.

rows = [                     # (subset, strategy, asked, correct)
    ("easy", "A", 1000, 950),
    ("easy", "B",  100,  97),
    ("hard", "A",  100,  40),
    ("hard", "B", 1000, 450),
]

totals = {}
for subset, strategy, n, c in rows:
    a, b = totals.get(strategy, (0, 0))
    totals[strategy] = (a + n, b + c)
    print(f"{subset:5} {strategy}  {c}/{n} = {c/n:.3f}")

for strategy, (n, c) in totals.items():
    print(f"overall {strategy}: {c}/{n} = {c/n:.3f}")

# easy  A  950/1000 = 0.950
# easy  B    97/100 = 0.970
# hard  A    40/100 = 0.400
# hard  B  450/1000 = 0.450
# overall A: 990/1100 = 0.900
# overall B: 547/1100 = 0.497

The overall number is measuring question difficulty, not strategy, because difficulty was not held constant between the arms. This is Simpson's paradox, and the reason to know it by name is that the reversal can go either way: the same imbalance can manufacture an improvement just as easily as it destroyed one here. Any aggregate over a mixture can be reversed by changing the mixture.

Controls that match the treatment#

A control arm is only a control if it differs from the treatment in exactly one respect. In practice several things move together, because the new method arrived with a new codebase, a new data loader and a different default. The discipline is to write down what differs before running anything, and to make the list as short as one line.

what must match                        why it is usually violated
the evaluation set, exactly            the new arm was tested on newer data
the training budget                    the new method "needed" longer to converge
the tuning effort                      the method got a sweep, the baseline defaults
the preprocessing                      the new pipeline also fixed a data bug
the random seeds                       one arm got lucky, and nobody checked
the metric and its averaging           macro in one table, micro in another
the person and the week                enthusiasm and skill are not constant

Pre-registering your analysis#

Pre-registration sounds like bureaucracy and is really a defence against your own cleverness. Before the runs finish, write down the metric, the subgroups you will report, how many seeds, and what result would count as the method failing. It takes five minutes, and it converts the most common failure — trying analyses until one is favourable — into something you would notice yourself doing.

• The primary metric, chosen now, and the threshold that counts as success.
• The subgroups you will break the result down by, listed in advance so they cannot be chosen afterwards.
• Number of seeds, and the spread you expect from them.
• What you will do if the result is ambiguous, which is the most likely outcome.
• What result would make you abandon the method. If there is none, you are not running an experiment.

It does not forbid looking further. Exploratory analysis is how findings are made, and the rule is only that it is labelled: this was the planned comparison, and these are things we noticed afterwards and would need to confirm on fresh data. Reviewers and interviewers both accept exploratory results readily when they are named as such, and distrust an entire paper when they suspect a planned result was chosen after the fact.

Negative results#

Most things you try will not work, and the record of what failed is worth more than it is given credit for — to you in six months, to the next person, and in an interview, where 'what have you tried that did not work' is a standard question and a vague answer is a poor signal. Keep the failed runs with enough detail to be interpretable: what you tried, what you expected, what happened, and your best guess at why.

The distinction that makes a negative result useful is between a method that does not work and an experiment that could not have detected it working. A run where the deltas were smaller than the seed spread has not shown the method fails; it has shown the experiment was underpowered, which is a different sentence and a much more honest one. Recording which of the two you have is most of the value.