Improving against a number

Improving a system against a number is the only way to know you are improving it, and it has a specific failure that catches careful people: run enough experiments against the same evaluation set and the score rises whether or not anything got better. This section is the loop — baseline, one change, measure, keep or revert — and the discipline that stops it deceiving you, with the deception measured so you can see how large it is.

The loop#

1. 1.Measure a baseline and write the number down before changing anything.
2. 2.Make exactly one change.
3. 3.Re-run the same evaluation on the same examples.
4. 4.Keep the change if it helped by more than the noise, revert it otherwise.
5. 5.Record what you tried and what happened, including the reverts.

Step five is the one that feels like overhead and is not. The record of failed changes is what stops the team trying them again in three months, and it is also the honest history of where the gains came from — which someone will ask for the moment the system is questioned.

One change at a time#

Changing the prompt and the retrieval settings together and seeing the score rise tells you the pair helped. It does not tell you that both did, and it is entirely possible that one helped by more than the pair and the other hurt. That is not a hypothetical — the experiment-design topic measures exactly this case, where two changes together look like a modest gain and separating them shows one was worth twice as much alone and the other was actively harmful.

The temptation to batch changes comes from each evaluation run costing money and minutes. The answer is to make the evaluation cheap enough to run often, not to batch the changes: a smaller sample for the inner loop and the full set at the end is a reasonable compromise, and batching is not.

What repeated measurement does to your eval set#

Here is the failure that gives this section its point. Twenty changes are tried against a hundred-example evaluation set, and every one of them is worthless — the underlying quality never moves. Each time a change scores better than the current best, it is kept, which is exactly what the loop above says to do.

import numpy as np
rng = np.random.default_rng(0)

TRUE = 0.70                    # the real quality. Nothing below changes it.
n_dev, n_held = 100, 2000

def evaluate(n):
    """Score on n examples. The change does nothing; only sampling noise moves."""
    return (rng.random(n) < TRUE).mean()

dev_best, kept = evaluate(n_dev), 0
for attempt in range(20):
    dev = evaluate(n_dev)
    if dev > dev_best:         # keep whatever helps, revert the rest
        dev_best, kept = dev, kept + 1

print("dev score after 20 changes:", round(dev_best, 3), f"({kept} kept)")
print("held-out score            :", round(evaluate(n_held), 3))
print("true quality              :", TRUE)

# dev score after 20 changes: 0.82 (5 kept)
# held-out score            : 0.705
# true quality              : 0.7

This is not an argument against measuring. It is an argument about how large the effect is, which depends on two things you control: how many examples the set has, and how many times you look at it.

def illusion(n_dev, attempts, trials=400, true=0.70):
    """Average apparent gain from keeping whatever helps, when nothing helps."""
    rng = np.random.default_rng(0)
    gains = []
    for _ in range(trials):
        best = (rng.random(n_dev) < true).mean()
        for _ in range(attempts):
            s = (rng.random(n_dev) < true).mean()
            if s > best:
                best = s
        gains.append(best - true)
    return float(np.mean(gains))

for n in (50, 100, 200, 500, 1000):
    print(n, round(illusion(n, 20) * 100, 1))

#   50  +11.7 points of pure noise
#  100   +8.4
#  200   +6.0
#  500   +4.0
# 1000   +2.8

for attempts in (5, 20, 50, 200):     # with a 200-example set
    print(attempts, round(illusion(200, attempts) * 100, 1))

#    5   +4.1
#   20   +6.0
#   50   +7.1
#  200   +8.6

Train and test splits#

The defence is the one every other part of machine learning already uses, applied to prompts and pipelines rather than to weights. Split your examples in two. The development set is the one you iterate against, look at freely and let yourself overfit — that is what it is for. The held-out set is touched only to answer the question that matters: is the system actually better than it was.

development set     look at it constantly    expect it to be optimistic
held-out set        touch it rarely          the number you report
production traces   growing continuously     where new examples come from

Knowing when to stop#

Hill climbing ends when the gains stop being real, and the signals are recognisable. Changes have started producing differences smaller than the run-to-run noise. The wins are arriving on the development set and not surviving on the held-out set. The remaining failures are all different from each other, which means there is no common cause left to fix. And the cheapest remaining change costs more than the problem it solves.

At that point the useful move is not another change to the same thing, but a different question — a harder set of examples, a different failure category, or an honest look at whether the metric still reflects what the product needs. Continuing to hill climb past this point is where the score keeps rising and the users stop noticing, which is the specific way that eval-driven development goes wrong.