Evals in the loop

An eval that has to be run by hand gets run when someone already suspects a problem, which is the one situation where it tells you least. This section is about wiring it into the ordinary flow of changes so it runs whether or not anyone is worried, and about what it should report: cost and latency beside quality, since changes routinely trade one against the others. It also covers keeping the set alive by feeding real failures back into it.

Automate it or it will not happen#

An eval you run manually gets run when you remember, which is when you already suspect a problem. The value is in catching the regression you did not suspect, and that only happens if it runs on every change.

# in CI, on every pull request
python -m evals.run --set core --compare-to main

# fails the build if the score drops more than 2 points

The arithmetic decides the design, so do it once with your own numbers rather than arguing about it. Cases, cost per case, seconds per case, how much you run in parallel, and how many pull request runs a day the team actually produces.

CASES_FULL, CASES_SMOKE = 240, 40
COST_PER_CASE = 0.006          # input + output tokens for one case, in dollars
SECONDS_PER_CASE = 2.4
CONCURRENCY = 8
PR_RUNS_PER_DAY = 45

def run_cost(n):
    return n * COST_PER_CASE, n * SECONDS_PER_CASE / CONCURRENCY

for name, n, per_day in (("smoke, every push", CASES_SMOKE, PR_RUNS_PER_DAY),
                         ("full, nightly", CASES_FULL, 1),
                         ("full, every push", CASES_FULL, PR_RUNS_PER_DAY)):
    cost, secs = run_cost(n)
    print(f"{name:<20} {cost:>6.2f} usd  {secs:>5.0f}s per run   "
          f"{cost * per_day * 30:>8.2f} usd per month")

# smoke, every push      0.24 usd     12s per run     324.00 usd per month
# full, nightly          1.44 usd     72s per run      43.20 usd per month
# full, every push       1.44 usd     72s per run    1944.00 usd per month

What goes in the smoke tier is a real decision and not a random sample. Forty cases chosen at random are mostly cases that have passed every day for a year and will pass again, which is forty cases of nothing. Choose instead by what has ever moved.

• Every case that has changed verdict in the last three months. These are the cases near the edge of what the system can do, and they move first.
• One case per failure mode from your error analysis, so a resurrected bug is caught by name.
• The cheapest case in each category, so no category is silently unrepresented.
• The two or three cases that cover anything expensive or irreversible: money moved, data deleted, an email sent.
• Everything that is a code check rather than a judge call, since those are free and fast. There is no reason to sample them at all.

The nightly tier runs everything, against the full set and with more runs per case, and it is where the number you quote comes from. The pull request tier exists only to stop obviously broken changes from merging, and it should be understood by everyone as a smoke alarm rather than a measurement.

Score is not the only output#

Report cost and latency alongside quality, every time. Changes frequently trade one for another, and a quality-only report hides that completely.

                 before    after
pass rate        84%       86%      better
mean cost        $0.031    $0.074   2.4x worse
p95 latency      2.1s      5.8s     worse

Better, if you can afford it. That is a decision, not a detail.

The gate: deciding what fails the build#

A gate has two failure modes and they pull in opposite directions. Too tight and it fires on clean changes, the team learns to re-run it until it passes, and within a month it is advisory. Too loose and a real regression merges. Both are measurable before you choose, by simulating the gate against a system that has not changed and against one that has.

import numpy as np
from math import comb

def exact_paired_p(down, up):
    """Sign test on the cases that changed verdict, in either direction."""
    n, k = down + up, max(down, up)
    if n == 0:
        return 1.0
    tail = sum(comb(n, i) for i in range(k, n + 1)) / 2 ** n
    return min(1.0, 2 * tail)

N = 120
BASE = 0.86

def one_run(true_rate, flip, rng):
    """Baseline and candidate on the same N cases. flip is the share of cases
    whose output is not stable, so they get re-rolled at the new true rate."""
    a = rng.random(N) < BASE
    b = a.copy()
    move = rng.random(N) < flip
    b[move] = rng.random(move.sum()) < true_rate
    return a, b

def gate_rates(true_rate, flip, trials=3000, seed=0):
    rng = np.random.default_rng(seed)
    fires = np.zeros(3)
    for _ in range(trials):
        a, b = one_run(true_rate, flip, rng)
        drop = a.mean() - b.mean()
        down = int((a & ~b).sum())
        up = int((~a & b).sum())
        fires += [drop > 0,
                  drop > 0.05,
                  exact_paired_p(down, up) < 0.05 and down > up]
    return fires / trials

print("            any drop   > 5 points   paired test")
no_change = gate_rates(BASE, 0.25)
print(f"no change   {no_change[0]:>8.0%} {no_change[1]:>11.0%} {no_change[2]:>12.0%}   <- false alarms")
for true_rate, label in ((0.78, "8 pt worse"), (0.66, "20 pt worse")):
    r = gate_rates(true_rate, 1.0)
    print(f"{label:<12}{r[0]:>8.0%} {r[1]:>11.0%} {r[2]:>12.0%}   <- caught")

#             any drop   > 5 points   paired test
# no change        42%          2%           1%   <- false alarms
# 8 pt worse       94%         74%          31%   <- caught
# 20 pt worse     100%        100%          95%   <- caught

Read the columns as three different products. Any-drop catches everything and cries wolf 42% of the time. A five-point threshold is the compromise that works at this size: it fires on 2% of clean runs and catches three quarters of an eight-point regression and all of a large one. The paired test is the most rigorous and the least useful here, because at 120 cases it demands more evidence than a moderate regression provides, and a gate that misses two thirds of real eight-point regressions is not doing its job.

So the honest reading is that at 120 cases you cannot have both. If an eight-point regression must be caught reliably, the answer is not a cleverer test, it is a bigger nightly set. Set the pull request threshold where the false alarms are rare, accept that it only catches the large breaks, and let the nightly run on a larger set with a paired comparison be the thing that catches the rest.

When the eval itself is flaky#

Before blaming a change, find out what the suite does when nothing changes at all. Run it ten times against the same commit. The spread you see is the floor on every comparison you will ever make with it, and it is usually larger than the effects people are arguing about.

import numpy as np
rng = np.random.default_rng(12)

N, RUNS = 120, 10
stable = rng.random(N)
# Each case has its own chance of passing. Most sit near 0 or 1; a few in the
# middle, and those are the ones that decide the score from run to run.
p = np.where(stable < 0.70, 0.99, np.where(stable < 0.88, 0.02, rng.uniform(0.2, 0.8, N)))

results = rng.random((RUNS, N)) < p          # RUNS x N pass/fail, system unchanged
scores = results.mean(axis=1)
print("ten runs of an unchanged system:", " ".join(f"{s:.3f}" for s in scores))
print(f"spread {100 * (scores.max() - scores.min()):.1f} points")

passes = results.sum(axis=0)
minority = np.minimum(passes, RUNS - passes)  # how often the case disagreed with itself
flaky = minority >= 2
print(f"{int(flaky.sum())} of {N} cases gave both verdicts at least twice in ten runs")

stable_only = results[:, ~flaky].mean(axis=1)
print(f"on the {int((~flaky).sum())} stable cases: spread "
      f"{100 * (stable_only.max() - stable_only.min()):.1f} points")

# ten runs of an unchanged system: 0.717 0.733 0.758 0.692 0.767 0.733 0.733 0.742 0.750 0.750
# spread 7.5 points
# 19 of 120 cases gave both verdicts at least twice in ten runs
# on the 101 stable cases: spread 4.0 points

What to do with those nineteen cases depends on why they flip, and the answer is always to read them rather than to delete them. A case is flaky for one of three reasons, and only one of them is the case's fault.

• The grader is ambiguous. Two outputs that a person would call equally good land on opposite sides of it. Fix the grader; this is the most common cause and the one worth the time.
• The system genuinely fails this input half the time. The case is telling the truth and the product is the problem. Move it out of the gate and into a reliability report, which is the same distinction the agent evaluation section draws.
• The judge is inconsistent with itself. Measure that directly by judging the same output twice, and if it is high, the rubric needs work before anything else is worth doing.

Quarantine is the holding pen, not the fix. Keep quarantined cases running and reported, with their flip rate visible, and keep them out of the gate until someone has decided which of the three they are. A quarantine list that never shrinks is a team that has stopped doing error analysis.

Making a non-deterministic suite behave#

Several small habits remove noise that has nothing to do with the system under test, and they are worth applying before you start enlarging sets.

• Set temperature to zero for graded runs. It does not make output identical on a hosted model, but it removes the largest source of variation for free.
• Fix the case order and any sampling seed, so a run is reproducible from its recorded configuration. Store the seed with the result.
• Pin the model version explicitly rather than taking the moving alias, and record it in the run. A silent model update looks exactly like a regression, and this is how you tell them apart.
• Cache by the hash of prompt, model and parameters. Unchanged cases then cost nothing, which is what makes running the suite on every push affordable.
• Run every case the same number of times and average. Three runs per case costs three times as much and cuts the standard error of the score by nearly half.
• Keep the judge's temperature, rubric and version in the same pinned configuration as the model. It is part of the instrument.

Keep the history#

Store every run. Direction over months tells you more than any single score, and it is what lets you answer whether the system is improving or just changing.

Store enough per run that a surprise is explainable six weeks later: the commit, the model version, the set version, the judge version, the seed, the per-case verdicts and the cost. The per-case verdicts are the ones people leave out and then need, because the useful question after a drop is never what is the score, it is which cases changed. Answering that from stored results takes a second; rerunning last month's system to find out takes an afternoon and often is not possible at all, because the model version is gone.

Feed production back in#

Sample real traffic into the eval set as the product evolves, and add every genuine failure as a case. Over time the set stops being something you assembled and becomes a description of what the system is actually asked to do.

Make it a rule with a step attached, in the way a good bug process does. Any incident, any complaint, any thumbs-down that turns out to be real, becomes a case in the set before the fix merges, and the case goes in failing. A fix that cannot be expressed as a case that was red and is now green is a fix nobody can defend in three months, and it is the single habit that most reliably turns an eval suite from a thing that was built into a thing that is used.