Eval-driven development

Improving a system against a number is the only way to know you are improving it, and it has a specific failure that catches careful people: run enough experiments against the same evaluation set and the score rises whether or not anything got better. This section is the loop — baseline, one change, measure, keep or revert — and the discipline that stops it deceiving you, with the deception measured so you can see how large it is.

The loop#

1. 1.Measure a baseline and write the number down before changing anything.
2. 2.Make exactly one change.
3. 3.Re-run the same evaluation on the same examples.
4. 4.Keep the change if it helped by more than the noise, revert it otherwise.
5. 5.Record what you tried and what happened, including the reverts.

Step five is the one that feels like overhead and is not. The record of failed changes is what stops the team trying them again in three months, and it is also the honest history of where the gains came from — which someone will ask for the moment the system is questioned.

One change at a time#

Changing the prompt and the retrieval settings together and seeing the score rise tells you the pair helped. It does not tell you that both did, and it is entirely possible that one helped by more than the pair and the other hurt. That is not a hypothetical — the experiment-design topic measures exactly this case, where two changes together look like a modest gain and separating them shows one was worth twice as much alone and the other was actively harmful.

The temptation to batch changes comes from each evaluation run costing money and minutes. The answer is to make the evaluation cheap enough to run often, not to batch the changes: a smaller sample for the inner loop and the full set at the end is a reasonable compromise, and batching is not.

What repeated measurement does to your eval set#

Here is the failure that gives this section its point. Twenty changes are tried against a hundred-example evaluation set, and every one of them is worthless — the underlying quality never moves. Each time a change scores better than the current best, it is kept, which is exactly what the loop above says to do.

import numpy as np
rng = np.random.default_rng(0)

TRUE = 0.70                    # the real quality. Nothing below changes it.
n_dev, n_held = 100, 2000

def evaluate(n):
    """Score on n examples. The change does nothing; only sampling noise moves."""
    return (rng.random(n) < TRUE).mean()

dev_best, kept = evaluate(n_dev), 0
for attempt in range(20):
    dev = evaluate(n_dev)
    if dev > dev_best:         # keep whatever helps, revert the rest
        dev_best, kept = dev, kept + 1

print("dev score after 20 changes:", round(dev_best, 3), f"({kept} kept)")
print("held-out score            :", round(evaluate(n_held), 3))
print("true quality              :", TRUE)

# dev score after 20 changes: 0.82 (5 kept)
# held-out score            : 0.705
# true quality              : 0.7

This is not an argument against measuring. It is an argument about how large the effect is, which depends on two things you control: how many examples the set has, and how many times you look at it.

def illusion(n_dev, attempts, trials=400, true=0.70):
    """Average apparent gain from keeping whatever helps, when nothing helps."""
    rng = np.random.default_rng(0)
    gains = []
    for _ in range(trials):
        best = (rng.random(n_dev) < true).mean()
        for _ in range(attempts):
            s = (rng.random(n_dev) < true).mean()
            if s > best:
                best = s
        gains.append(best - true)
    return float(np.mean(gains))

for n in (50, 100, 200, 500, 1000):
    print(n, round(illusion(n, 20) * 100, 1))

#   50  +11.7 points of pure noise
#  100   +8.4
#  200   +6.0
#  500   +4.0
# 1000   +2.8

for attempts in (5, 20, 50, 200):     # with a 200-example set
    print(attempts, round(illusion(200, attempts) * 100, 1))

#    5   +4.1
#   20   +6.0
#   50   +7.1
#  200   +8.6

Train and test splits#

The defence is the one every other part of machine learning already uses, applied to prompts and pipelines rather than to weights. Split your examples in two. The development set is the one you iterate against, look at freely and let yourself overfit — that is what it is for. The held-out set is touched only to answer the question that matters: is the system actually better than it was.

development set     look at it constantly    expect it to be optimistic
held-out set        touch it rarely          the number you report
production traces   growing continuously     where new examples come from

Knowing when to stop#

Hill climbing ends when the gains stop being real, and the signals are recognisable. Changes have started producing differences smaller than the run-to-run noise. The wins are arriving on the development set and not surviving on the held-out set. The remaining failures are all different from each other, which means there is no common cause left to fix. And the cheapest remaining change costs more than the problem it solves.

At that point the useful move is not another change to the same thing, but a different question — a harder set of examples, a different failure category, or an honest look at whether the metric still reflects what the product needs. Continuing to hill climb past this point is where the score keeps rising and the users stop noticing, which is the specific way that eval-driven development goes wrong.

Everything in this topic has been described one piece at a time: traces, failure categories, evals, held-out splits, one change at a time. This section runs them together on a single feature across five days, with the numbers at each step, including the two changes that were measured and thrown away. It is here because the order of the work is the part that is hard to convey in the abstract, and the order is what makes it work.

The setup#

A support assistant answers questions about a software product from its help centre. The team says it 'feels unreliable'. Here is a realistic week, condensed.

Monday: read before measuring#

Read 120 recent traces and write a note on the first failure in each. Group the notes.

Answered from outdated article          17
Follow-up question lost its context     12
Invented a setting that does not exist   8
Too long, buried the answer              6
Other                                    5
                                        --
                                        48 failing of 120 (40%)

Tuesday: build the eval around those modes#

• 80 cases from real traffic, including all 48 failures, split into 56 to tune on and 24 held out.
• Code check for outdated sources: fail if any cited article is marked archived.
• Pass or fail judge for invented settings, validated on 40 labels: catches 17 of 19 failures, accepts 20 of 21 passes.
• Baseline: 22 of 56 on the tune set and 10 of 24 held out. That is 32 of 80, which matches the 48 known failures in the set.

Wednesday and Thursday: one change at a time#

Change                                         Tune set   Kept?
Exclude archived articles from the index         36/56     yes
Rewrite follow-ups into standalone questions     42/56     yes
Add 'only mention settings from the sources'     44/56     yes
Few-shot examples of short answers               45/56     no, +40% cost for +1 case
Switch to a larger model                         46/56     no, 3x cost

The first two changes were not prompt changes at all. The biggest wins in a system usually come from data and retrieval, which is why reading traces first matters.

Friday: check the held-out set once#

Held-out result: 17 of 24 pass, up from 10. The tune set went from 22 to 44 of 56 over the same changes. Both moved a lot, which suggests the changes generalise. If the tune set had jumped and the held-out set had barely moved, that would point to overfitting the tuning cases.

Writing it up#

What we found: 40% of traces failed; top causes were outdated articles
  and lost follow-up context.
What we changed: excluded archived articles, added standalone rewriting,
  constrained settings to sources.
Result: held-out pass rate 10/24 -> 17/24. Cost per answer +8%, p95 +300 ms.
Not done: answer length (tried few-shot, not worth the cost).
Next: the eval set is small; add 100 cases from next week's traffic.