Is the difference real? Statistics for evals

Eval scores move about on their own, and without a sense of how much, you cannot tell an improvement from a coincidence. This section is the minimum statistics for the job: putting an error bar on a pass rate, comparing two versions on the same cases so the comparison is tighter, allowing for the model's own run-to-run variation, and working out how many cases you need before a number means anything. It is the difference between measuring a change and believing in one.

The problem#

Version A passes 40 of 50 cases. Version B passes 42. Is B better? Almost certainly you cannot tell. Teams ship changes like this every week and call them improvements. Knowing how much a score can wobble by chance, and so whether a difference is statistically significant, is what separates measurement from wishful thinking.

A pass rate has an error bar#

A pass rate from a sample is an estimate. With few cases the range of true values consistent with it is wide. The Wilson interval is a good default for pass rates, and it behaves sensibly near 0% and 100% where the simple formula does not.

from math import sqrt

def wilson(passed, n, z=1.96):
    p = passed / n
    denom = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / denom
    half = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / denom
    return centre - half, centre + half

print(wilson(40, 50))    # about (0.67, 0.89)
print(wilson(400, 500))  # about (0.76, 0.83)

At fifty cases, 80% could plausibly be anything from the high sixties to the high eighties. Ten times the cases narrows that to a few points either side.

Compare on the same cases#

When both versions run on the same set, most cases pass or fail for both. Only the cases where they disagree carry information about which is better. Count them.

                 B passes   B fails
A passes            38          2
A fails              4          6

Disagreements: B won 4, A won 2.

Six disagreements split four to two is the kind of result you get from a coin tossed six times. A simple test on the disagreements alone, known as McNemar's test, formalises this. With so few disagreements it will say what intuition should: no evidence that B is better.

from math import comb

def paired_p(b_wins, a_wins):
    """Two-sided exact test on the disagreeing cases only."""
    n, k = b_wins + a_wins, max(b_wins, a_wins)
    tail = sum(comb(n, i) for i in range(k, n + 1)) / 2 ** n
    return min(1.0, 2 * tail)

print(paired_p(4, 2))   # about 0.69, no evidence either way
print(paired_p(18, 4))  # about 0.004, real difference

One number is not a result#

Everything above treats the eval as a single pass rate, which is how it gets reported and how the most expensive mistakes get through. A set covers several categories, a change rarely affects them equally, and the aggregate is a weighted average that can rise while the part you care about falls. Print the breakdown every time, with an interval per category, and read that before the headline.

# (category, cases, passed before, passed after)
RESULTS = [
    ("track an order", 120, 108, 114),
    ("change a plan",   60,  49,  53),
    ("invoice query",   40,  30,  35),
    ("account access",  30,  23,  25),
    ("refund request",  50,  40,  32),
]

print(f"{'category':<16}{'n':>4}{'before':>8}{'after':>7}{'change':>8}   95% intervals")
tot_n = tot_b = tot_a = 0
for name, n, before, after in RESULTS:
    tot_n, tot_b, tot_a = tot_n + n, tot_b + before, tot_a + after
    lo_b, hi_b = wilson(before, n)
    lo_a, hi_a = wilson(after, n)
    print(f"{name:<16}{n:>4}{before / n:>8.0%}{after / n:>7.0%}{(after - before) / n:>+8.0%}"
          f"   before {lo_b:.0%}-{hi_b:.0%}  after {lo_a:.0%}-{hi_a:.0%}")
print(f"{'aggregate':<16}{tot_n:>4}{tot_b / tot_n:>8.0%}{tot_a / tot_n:>7.0%}"
      f"{(tot_a - tot_b) / tot_n:>+8.0%}")

# category           n  before  after  change   95% intervals
# track an order   120     90%    95%     +5%   before 83%-94%  after 90%-98%
# change a plan     60     82%    88%     +7%   before 70%-89%  after 78%-94%
# invoice query     40     75%    88%    +12%   before 60%-86%  after 74%-95%
# account access    30     77%    83%     +7%   before 59%-88%  after 66%-93%
# refund request    50     80%    64%    -16%   before 67%-89%  after 50%-76%
# aggregate        300     83%    86%     +3%

This is not a rare pathology. It is what you should expect whenever a change helps the common path, since the common path dominates the average by construction. A prompt that adds a summarising instruction improves four categories of question and breaks the one where the detail mattered, and the aggregate says yes.

Two working rules follow. Report per category always, and make the gate look at categories rather than the total, so a large drop in any one of them fails the build even when the headline rose. And accept what the intervals in that table are saying: a thirty-case category carries a range of roughly fifteen points either side, so per-category numbers need per-category sample sizes. If a category matters enough to gate on, it needs enough cases to measure, which in practice means at least thirty and preferably more.

The set moved, not the system#

The other way an aggregate lies is when the denominator changes underneath it. Cases get added every month, which is correct practice, and each addition shifts the mix. Then two numbers that were never comparable get compared in a slide.

# Same system, same behaviour on every case it already had. All that changed
# is that 60 cases from an easy category were added to the set in March.
feb = {"n": 300, "passed": 249}
mar = {"n": 360, "passed": 249 + 57}
print(f"February  {feb['passed']}/{feb['n']} = {feb['passed'] / feb['n']:.0%}")
print(f"March     {mar['passed']}/{mar['n']} = {mar['passed'] / mar['n']:.0%}")
print(f"reported improvement {100 * (mar['passed'] / mar['n'] - feb['passed'] / feb['n']):+.1f} points")
print(f"on the 300 cases both months share: {feb['passed'] / feb['n']:.0%} then {249 / 300:.0%}")

# February  249/300 = 83%
# March     306/360 = 85%
# reported improvement +2.0 points
# on the 300 cases both months share: 83% then 83%

So version the set, re-baseline when it changes, and when you plot a score over months, plot it on the cases that existed for the whole period rather than on whatever the set contained that week. The same discipline, applied to a set you tune against repeatedly rather than one you add to, is what the eval-driven development topic measures under improving against a number: twenty worthless changes against a hundred cases produce twelve points of improvement that is not there.

The model varies too#

Run the same version twice and the score changes, because sampling and hosted inference are not perfectly repeatable. Before comparing versions, run one version three to five times and look at the spread. If it moves by three points between identical runs, a two point improvement is noise.

How many cases you need#

• To catch large regressions, such as a feature breaking, twenty to fifty cases is enough.
• To detect a change of around five points reliably, you need several hundred cases, or a paired comparison with enough disagreements.
• Small improvements of one or two points need thousands of cases. Most teams cannot measure these, and should not claim them.

Those figures are worked out with a simulation in the product judgement topic, under choosing and changing models, where a genuine four-and-a-half point improvement is called correctly by fifty examples only one time in five. It is the same arithmetic in a different setting, and running it once on your own numbers is the fastest way to stop arguing about whether a two-point move is real.

One habit undoes all of this, and it is the one every team drifts into: running the eval, seeing no difference, changing something small, running it again, and stopping when a run finally looks good. Each look is another chance for noise to produce a winner, so the false-positive rate is not the 5% you think you are working to. The experiment design topic counts exactly this, and the numbers are worth carrying here: checking once gives 4.9% false positives, checking fifty times gives 32.1%. Decide the sample size and the criterion before the run, and let the run finish.