Evaluation

Everything else in this topic rests on being able to answer one question: did that change make the system better or worse? This section is about why the question is hard for anything built on a model, why the testing tools you already have do not answer it, and what takes their place. It is worth taking seriously because the alternative compounds: a few months of unmeasured prompt edits produces a system nobody on the team is willing to touch.

The question you are actually asking             What answers it

"did my change break anything?"                 a regression set and a gate
"which of these two prompts is better?"         both on the same cases, paired
"is this good enough to ship?"                  a set shaped like real traffic
"how often does failure mode X happen now?"     a targeted set, one grader for X
"can I trust the grader itself?"                your own labels, and a kappa
"is this agent reliable?"                       the same task run k times

The problem#

You change a prompt. Is the system better? With ordinary code you run the tests. With a model you look at a couple of outputs, they seem fine, and you ship. That is not a measurement, and over a few months it produces a system nobody can safely change.

An eval is the missing measurement: a fixed set of inputs, a way of scoring the output, and a number you can compare between versions.

Why unit tests do not cover it#

Model output varies between runs, even at temperature zero on a hosted API, and there is rarely one correct string. Asserting equality against expected text produces a suite that flakes, gets ignored, and is eventually deleted.

import random

article = "The company reported record profits of 4.1bn for the year."

def summarise(text):
    """Stands in for a model call: same meaning, different words each time."""
    return random.choice(["The company reported record profits.",
                          "Record profits were reported by the company.",
                          "The company posted record profits of 4.1bn."])

def judge_says_accurate(summary, source):
    return "profit" in summary.lower() and "loss" not in summary.lower()

# Will flake, then be deleted
random.seed(0)
try:
    assert summarise(article) == "The company reported record profits."
    print("passed, this time")
except AssertionError:
    print("failed, and the summary was fine")

# Stable, and actually meaningful
for seed in range(5):
    random.seed(seed)
    result = summarise(article)
    assert len(result.split()) < 50
    assert "profit" in result.lower()
    assert judge_says_accurate(result, article)
print("5 seeds, 3 different summaries, all pass")

# failed, and the summary was fine
# 5 seeds, 3 different summaries, all pass

Where most teams are#

• Nobody can say whether last month's prompt change helped.
• A regression is discovered by a user, not by CI.
• Prompt edits are made by whoever is least afraid to touch it.
• Nobody will change the prompt before a demo.

Start small, today#

Twenty examples in a file and a script that scores them is enough to change how a project runs. It does not need a framework and it does not need to be complete before it is useful.

CASES = [
    {"input": "...", "must_contain": ["refund", "5 days"]},
    {"input": "...", "must_be_json": True},
    {"input": "...", "expect_empty": True},
]

The whole harness is about thirty lines. A case is an input, a check, and an id you can say out loud in a stand-up. The check is ordinary Python: the cheapest useful eval is a list of predicates, and you can add a judge later for the cases where no predicate works.

import json

CASES = [
    {"id": "refund-annual", "input": "refund on the annual plan after 20 days?",
     "check": lambda out: "14 days" in out and "30" not in out},
    {"id": "empty-source", "input": "what is the refund policy for the beta tier?",
     "check": lambda out: "do not have" in out or "not sure" in out},
    {"id": "json-shape", "input": "list my open orders as json",
     "check": lambda out: isinstance(json.loads(out), list)},
]

def system(text):
    """Stands in for the thing you are evaluating."""
    if "annual" in text:
        return "Annual plans can be refunded within 14 days."
    if "beta" in text:
        return "Beta tier refunds are handled within 30 days."
    return '[{"id": "A-101"}]'

def run(cases):
    rows = []
    for case in cases:
        try:
            out = system(case["input"])
            ok = bool(case["check"](out))
            note = ""
        except Exception as e:                 # a crash is a failure, not an error
            out, ok, note = "", False, type(e).__name__
        rows.append((case["id"], ok, note, out))
    passed = sum(ok for _, ok, _, _ in rows)
    print(f"{passed}/{len(rows)} passed")
    for cid, ok, note, out in rows:
        if not ok:
            print(f"  FAIL {cid} {note} -> {out[:60]!r}")
    return passed / len(rows)

run(CASES)

# 2/3 passed
#   FAIL empty-source  -> 'Beta tier refunds are handled within 30 days.'

Three things in that harness matter more than they look. An exception counts as a failure rather than crashing the run, because the eval you abandon halfway through is not a measurement. Every case has an id, so a regression is reported as a name and not a percentage. And the failures print with their output attached, since the first thing you do with a red eval is read what actually came back.

What the rest of this topic does with that number#

The sections after this one fill in the parts the thirty-line version leaves out, roughly in the order you will need them. Read traces first, so the cases are shaped like your real failures. Build the set from production rather than imagination. Choose a grader per case, and validate the grader when it is a model. Run the whole thing on every change. Then read the result honestly, which mostly means per category and with an error bar.

Three questions that sit next to evals are answered in neighbouring topics rather than here, and it is worth knowing where they are. How many examples you need before a comparison can decide anything is worked out with numbers in the product judgement topic, under choosing and changing models. What happens to a set you tune against repeatedly is measured in eval-driven development, under improving against a number. And the same statistics applied to real users, including what checking the dashboard repeatedly does to your false-positive rate, is the experiment design topic.

This is the first step of evaluation and almost everyone skips it, which is why so many eval suites produce tidy numbers about nothing in particular. The work is manual: sit with a sample of real runs, describe each failure in your own words, then group and count those descriptions until a ranked list of failure modes falls out. Everything you build afterwards is shaped by that list, so this section decides whether the rest of your evaluation is aimed at anything.

The mistake this section prevents#

Most people start evaluation by picking metrics: helpfulness, faithfulness, tone. They wire up a library that scores all three, get numbers that look reasonable, and learn nothing. The metrics were chosen before anyone looked at how the system actually fails, so they measure things that were never the problem.

The order that works is the reverse. Read what the system did, write down what went wrong in plain words, group those notes, count them, and only then decide what to measure. The evals you end up with are shaped like your real failures instead of like a generic checklist.

What a trace is#

A trace is the full record of one request: the user input, the system prompt, every model call with its exact input and output, every tool call with its arguments and result, and the final answer. If you only log the final answer you cannot tell whether a wrong reply came from bad retrieval, a bad tool result, or the model ignoring good context.

trace = {
    "id": "t_0192",
    "input": "Can I get a refund on the annual plan after 20 days?",
    "steps": [
        {"type": "retrieve", "query": "annual plan refund", "hits": ["policy#4", "faq#12"]},
        {"type": "llm", "prompt_version": "v7", "output": "Yes, within 30 days..."},
    ],
    "final": "Yes, you can get a full refund within 30 days.",
    "latency_ms": 2140,
    "cost_usd": 0.0031,
}

Three fields in there earn their place for reasons that are not obvious until you need them. The prompt version, because half of what you find will turn out to be one prompt revision that shipped on a Tuesday. The retrieval hits by id, because that is what separates the model ignoring the right document from the system never fetching it. And the cost, because a failure mode that is cheap and a failure mode that burns two dollars a run are different priorities even when they happen equally often.

What you cannot recover later is worth listing, since most teams discover it the week they start this work. Raw model output before any parsing or post-processing, because the bug is often in your parser. Tool arguments exactly as sent, not as rendered in a log line. The system prompt as it actually went out, variables filled in. Anything the user did afterwards: retried, rephrased, escalated, left. And a stable id per conversation, so you can find the trace when a customer quotes one sentence of it back to you.

Which hundred traces to read#

Not the first hundred, and not the hundred with the worst feedback. The first hundred are one hour of one day and carry whatever was happening that hour. The worst-rated hundred tell you about the failures users bother to report, which is a biased and small subset — most people just leave.

Take a stratified sample, the way the dataset section builds an eval set, and then add the obvious suspects on top as a separate pile. Keep the two piles apart in your notes, because they answer different questions: the stratified pile tells you how often things go wrong, and the suspect pile tells you how badly.

• A random draw within each intent or category, with a floor so rare categories are present at all.
• Every conversation the user escalated, retried three times, or abandoned mid-way.
• Every trace where a code check already fails: unparseable output, a citation that resolves to nothing, a tool error.
• The longest and the most expensive runs, which are where loops and thrashing hide.
• A handful of ordinary successful traces, so you calibrate on what normal looks like and do not start seeing failures everywhere.

Step one: open coding#

Pull a sample of traces, around a hundred to start. For each one, read it end to end and write one short note about the first thing that went wrong. Not a score. A sentence, in your own words, specific to that trace.

t_0192  Policy says 14 days for annual plans. Retrieved the monthly plan section instead.
t_0193  Fine.
t_0194  Answered in English, user wrote in German.
t_0195  Promised a refund the agent has no power to issue.
t_0196  Fine.
t_0197  Retrieved the right section, then quoted the wrong number from it.

Note only the first failure in each trace. Later errors are often caused by the first one, and counting them separately inflates problems that fix themselves.

Step two: group notes into failure modes#

Once you have notes, read them together and cluster the ones that describe the same underlying problem. Give each cluster a name that someone could act on. 'Bad answer' is not a failure mode. 'Retrieved the section for the wrong plan type' is.

Failure mode                              Count   Share
Retrieved the wrong plan's policy            14     31%
Promised actions the bot cannot take          9     20%
Quoted wrong number from correct source       7     16%
Wrong language                                6     13%
Refused a question it could answer            5     11%
Other                                         4      9%
                                             --
                                             45   failing of 100 traces

Two rules make that table stable rather than a matter of mood. A mode has to name what went wrong, not how bad it was, so that a fix is implied by the name. And a mode only exists once three traces belong to it; below that it stays in an 'other' pile you re-read at the end, when you usually find that four of them were the same thing described four ways.

Then weight the counts by what each one costs#

Counting is the right first move and the wrong last one. The most frequent mode is often a minor annoyance and the rarest is often the one that ends up in a regulator's inbox. Put a cost per occurrence beside the count, multiply by the traffic the sample was drawn from, and the ordering usually changes.

# Counted from 100 read traces, with what one occurrence costs the business.
MODES = [
    ("retrieved the wrong plan's policy", 14, 4.0),
    ("promised an action the bot cannot take", 9, 30.0),
    ("quoted the wrong number from the right source", 7, 12.0),
    ("replied in the wrong language", 6, 2.0),
    ("refused a question it could answer", 5, 3.0),
    ("leaked another customer's order id", 1, 900.0),
]
TRAFFIC = 40_000        # conversations a month

print(f"{'failure mode':<46}{'count':>6}{'cost':>8}{'per month':>12}")
rows = []
for name, count, unit in MODES:
    monthly = count / 100 * TRAFFIC * unit
    rows.append((monthly, name, count, unit))
for monthly, name, count, unit in sorted(rows, reverse=True):
    print(f"{name:<46}{count:>6}{unit:>8.0f}{monthly:>12,.0f}")

# failure mode                                   count    cost   per month
# leaked another customer's order id                 1     900     360,000
# promised an action the bot cannot take             9      30     108,000
# quoted the wrong number from the right source      7      12      33,600
# retrieved the wrong plan's policy                 14       4      22,400
# refused a question it could answer                 5       3       6,000
# replied in the wrong language                      6       2       4,800

Do not over-engineer the cost column. Support minutes, refunds issued, churn risk and regulatory exposure in whatever unit you can defend, rounded hard, agreed with whoever owns the product. Its job is to stop the loudest count from winning automatically, and a rough figure does that job.

Step three: decide what each mode needs#

Not every failure deserves an eval. Some are just bugs. If the wrong language comes from a missing instruction, add the instruction and move on. Build evals for failures that are likely to come back when someone edits the prompt, changes the model, or touches retrieval.

• Fix directly: a missing instruction, a broken tool, a template bug. Then add one regression case.
• Code check: anything a function can verify, such as the language of the reply or whether a number matches the source.
• Judge: anything that needs reading, such as whether the bot promised something it cannot do. One judge per failure mode, answering pass or fail.

If you have no users yet#

You still need traces. Generate realistic inputs deliberately rather than asking a model for 'fifty test questions', which gives you fifty variations of the same polite easy question. Pick the dimensions along which real inputs differ, list values for each, and combine them.

import itertools, random

personas  = ["new customer", "long-time customer", "angry customer", "non-native speaker"]
topics    = ["refund", "plan change", "invoice error", "account access"]
scenarios = ["clear question", "missing key detail", "asks for something not allowed", "two questions at once"]

combos = random.sample(list(itertools.product(personas, topics, scenarios)), 60)
# Turn each combo into a natural message: by hand for the first ten,
# then with a model, then read every one and delete the unrealistic ones.

Run the system on those inputs, save full traces, and do the same open coding. Replace synthetic traces with real ones as soon as you have any.

When to stop reading#

Keep reading until new traces stop producing new failure modes. When twenty traces in a row only add to categories you already have, you have seen most of what this sample can show you. Repeat the whole exercise after any large change, and on a small sample of production traffic every week.

That rule of thumb has a number behind it, and the number is easy. The share of your traces showing a failure mode you have seen exactly once estimates the chance that the next trace shows a mode you have never seen at all. It is called the Good-Turing estimate, it was invented for a code-breaking problem with the same shape, and it takes one line. Here is a simulated system with eighteen real failure modes, a few common and a long tail, so the estimate can be checked against a truth you would not normally have.

import numpy as np
rng = np.random.default_rng(5)

# A made-up system with 18 real failure modes, a few common and a long tail.
weights = np.array([1 / (i + 1) ** 1.1 for i in range(18)])
weights = weights / weights.sum()

def read_traces(n):
    """Which failure mode each of n failing traces turned out to show."""
    return rng.choice(len(weights), size=n, p=weights)

print(f"{'traces read':>12}{'modes found':>13}{'seen once':>11}{'new-mode rate':>15}{'truth':>8}")
for n in (20, 40, 80, 160, 320):
    sample = read_traces(n)
    counts = np.bincount(sample, minlength=len(weights))
    found = int((counts > 0).sum())
    singletons = int((counts == 1).sum())
    # Good-Turing: the share of traces showing a mode exactly once estimates
    # the chance the next trace shows a mode you have never seen.
    estimate = singletons / n
    truth = weights[counts == 0].sum()
    print(f"{n:>12}{found:>13}{singletons:>11}{estimate:>15.1%}{truth:>8.1%}")

#  traces read  modes found  seen once  new-mode rate   truth
#           20           10          6          30.0%   24.4%
#           40           12          4          10.0%   16.0%
#           80           16          4           5.0%    3.3%
#          160           18          3           1.9%    0.0%
#          320           18          0           0.0%    0.0%

Two honest limits. The estimate only counts modes you can distinguish, so it goes to zero the moment you stop looking carefully, and a lazy reader reaches saturation quickly and wrongly. And it says nothing about modes that cannot appear in this sample at all, such as the failures of a feature that launches next month. Treat it as a stopping rule for this round of reading, not a certificate that the system has eighteen problems.

One person who understands the domain should own the labels. A committee produces disagreements about what 'good' means and no labels. That person can be you.

That said, have a second person label thirty of the same traces once, early. You are about to build graders that encode these definitions, and if two careful people disagree about which mode a trace belongs to, no rubric written from those definitions can be consistent either. Measure that disagreement the same way the grading section measures a judge, with a kappa rather than a raw agreement percentage, and fix the definitions before writing a single grader. Half an hour spent here saves the week that otherwise goes into tuning a judge against labels that were never stable.

An eval is only as honest as the cases in it, and the natural way to build a set, sitting down and thinking up inputs, produces the one thing you do not want: a test your system passes. This section is about where cases actually come from, what has to be in the set for a score to mean anything, and how to keep that score comparable as the set grows. Get this wrong and you still get a number every week, it just will not track reality.

Where a case came from          What it is good for           How many

production failure              proving one bug is fixed      all of them
escalation or thumbs-down       the failures users noticed    all of them
random sample of traffic        the overall number            100-500
stratified sample by intent     per-category numbers          20+ per category
hand-written edge case          empty, huge, hostile input    10-30
model-generated input           filling a category you lack   as few as possible

Use real inputs#

Invented test cases are unconsciously shaped to be answerable, so they measure a system handling the inputs you imagined rather than the ones it gets. Pull from logs.

• Common cases, in roughly the proportion they occur.
• Every failure you have already seen in production. These are the highest-value cases you own.
• Edge cases: empty input, enormous input, wrong language, adversarial input.
• Cases where the correct answer is to refuse or to say nothing was found.

Sampling, or how the rare categories vanish#

Taking a hundred conversations at random from the log feels like the neutral choice and is not, because the categories that matter least by volume often matter most by consequence. Here is last month's traffic and what a random sample of a hundred actually contained.

import numpy as np
rng = np.random.default_rng(2)

# Share of last month's traffic, by what the user was asking for.
MIX = {
    "track an order": 0.41,
    "change a plan": 0.22,
    "refund request": 0.16,
    "invoice query": 0.11,
    "account access": 0.07,
    "cancel and complain": 0.03,
}
names = list(MIX)
weights = np.array(list(MIX.values()))
log = rng.choice(len(names), size=8000, p=weights)

def counts(sample):
    return {names[i]: int((sample == i).sum()) for i in range(len(names))}

random_100 = rng.choice(log, size=100, replace=False)
print("random 100 from the log")
for k, v in counts(random_100).items():
    print(f"  {k:<22}{v:>4}")

# Stratified: at least 8 of everything, the rest in proportion.
floor = 8
extra = 100 - floor * len(names)
plan = {n: floor + int(round(extra * w)) for n, w in MIX.items()}
print("stratified 100, floor of", floor)
for k, v in plan.items():
    print(f"  {k:<22}{v:>4}")
print("total", sum(plan.values()))

# random 100 from the log          stratified 100, floor of 8
#   track an order          47        track an order          29
#   change a plan           14        change a plan           19
#   refund request          20        refund request          16
#   invoice query           14        invoice query           14
#   account access           3        account access          12
#   cancel and complain      2        cancel and complain     10
#                                     total 100

A floor per category fixes that, at a price: the set no longer looks like traffic, so the headline number drifts away from what users experience. Weight the categories back to the real mix when you report it, and keep both figures. The gap between them is usually a few points and is entirely a property of how you sampled.

# A stratified set no longer looks like traffic, so weight the categories back.
passed = {"track an order": 28, "change a plan": 17, "refund request": 12,
          "invoice query": 11, "account access": 8, "cancel and complain": 5}
flat = sum(passed.values()) / sum(plan.values())
weighted = sum(MIX[n] * passed[n] / plan[n] for n in names)
print(f"unweighted mean of the stratified set {flat:.1%}")
print(f"weighted back to the traffic mix      {weighted:.1%}")

# unweighted mean of the stratified set 81.0%
# weighted back to the traffic mix      86.1%

A production failure is worth more than an invented case#

This is the claim worth testing rather than repeating. Suppose a bug affects 6% of traffic and you write a fix that works four times in five. A random eval set contains that bug only in proportion, so most of its cases are irrelevant to the question you are asking. Cases pulled from reports of the bug itself are all relevant. The difference is not a matter of taste.

import numpy as np
from math import comb
rng = np.random.default_rng(4)

def exact_paired_p(fixed, broken):
    """Two-sided sign test on the cases that changed verdict."""
    n, k = fixed + broken, max(fixed, broken)
    if n == 0:
        return 1.0
    tail = sum(comb(n, i) for i in range(k, n + 1)) / 2 ** n
    return min(1.0, 2 * tail)

def detects(n_cases, share_affected, fix_rate=0.80, break_rate=0.02, trials=4000):
    """How often does this set show the fix, at p < 0.05 on a paired test?"""
    wins = 0
    for _ in range(trials):
        affected = rng.random(n_cases) < share_affected
        fixed = int((affected & (rng.random(n_cases) < fix_rate)).sum())
        broken = int((~affected & (rng.random(n_cases) < break_rate)).sum())
        wins += exact_paired_p(fixed, broken) < 0.05 and fixed > broken
    return wins / trials

print("a bug in 6% of traffic, a fix that works 4 times in 5")
for n in (100, 200, 500, 1000):
    print(f"  random set of {n:>5}: detected {detects(n, 0.06):.0%} of the time")
for n in (20, 40):
    print(f"  {n} cases pulled from reports of that bug: {detects(n, 1.0):.0%}")

#   random set of   100: detected 9% of the time
#   random set of   200: detected 27% of the time
#   random set of   500: detected 67% of the time
#   random set of  1000: detected 95% of the time
#   20 cases pulled from reports of that bug: 100%
#   40 cases pulled from reports of that bug: 100%

It is also the argument for keeping the two sets apart. The targeted set answers did this fix work, and it will always say yes more loudly than reality, because every case in it was selected for being broken. The traffic-shaped set answers how much this matters overall, and it needs to be large. Report them separately and neither one lies to you.

Write the expected outcome first#

Decide what a good answer looks like before you run the system. If you look at the output first, you will write a criterion the current version happens to satisfy, and your eval will certify whatever you already have.

Which means a case is more than an input and a string. It carries where it came from, which category it belongs to, which grader decides it and what that grader is looking for. Write that down as a record from the first day, because retrofitting provenance onto four hundred cases is a week nobody has.

CASE = {
    "id": "refund-annual-day20",
    "input": "Can I get a refund on the annual plan after 20 days?",
    "category": "refund request",
    "source": "ticket 48211, escalated 2026-02-11",   # never 'made up by me'
    "grader": "contains_all",
    "expect": {"all_of": ["14 days"], "none_of": ["30 days", "no refund"]},
    "added": "2026-02-12",
    "set": "core-v4",
}

Growing it without filling it with copies#

Sampling straight from logs gives you the same question thirty times, because most traffic is the same question. Duplicates do not just waste money; they silently reweight the set, so an improvement on one popular phrasing looks like an improvement everywhere. Deduplicate before you add, on word overlap rather than string equality.

LOG = [
    "where is my order",
    "Where is my order?",
    "where is my order!!",
    "can you tell me where my order is",
    "my order hasn't arrived, where is it",
    "how do I change my plan",
    "how do i change my plan to annual",
    "I want to downgrade to the free tier",
    "the invoice says 1200 but I paid 1000",
    "why was I charged twice this month",
    "cancel my account and delete my data",
]

def shingles(text, k=3):
    """Sets of k consecutive words. Cheap, no model, no embedding service."""
    words = text.lower().replace("?", "").replace("!", "").replace(",", "").split()
    if len(words) < k:
        return {" ".join(words)}
    return {" ".join(words[i:i + k]) for i in range(len(words) - k + 1)}

def jaccard(a, b):
    return len(a & b) / len(a | b)

kept, dropped, sigs = [], [], []
for text in LOG:
    s = shingles(text)
    near = next((kept[i] for i, t in enumerate(sigs) if jaccard(s, t) >= 0.5), None)
    if near is None:
        kept.append(text)
        sigs.append(s)
    else:
        dropped.append((text, near))

print(f"{len(LOG)} log lines -> {len(kept)} distinct cases")
for text, near in dropped:
    print(f"  dropped {text!r}")
    print(f"     near {near!r}")

# 11 log lines -> 8 distinct cases
#   dropped 'Where is my order?'
#      near 'where is my order'
#   dropped 'where is my order!!'
#      near 'where is my order'
#   dropped 'how do i change my plan to annual'
#      near 'how do I change my plan'

Note what word overlap does not catch: it keeps the two order-tracking paraphrases that share almost no words, which is the right answer for a test set and the wrong answer if you were deduplicating for cost. Embeddings catch those too, and then start merging cases that differ only in the detail that matters. A category label maintained by hand beats both.

Split it#

This is borrowed from machine learning and applies exactly. If you tune prompts against the whole set, you overfit to it: the score climbs while real performance does not. Hold some cases back and never look at them while iterating.

train  (70%)   tune against these, look at them freely
test   (30%)   score only, never inspect while iterating

Report the test number. That is the one that predicts reality.

Split by category rather than at random, so both halves contain every category, and split by user or conversation rather than by case where several cases came from the same conversation. Otherwise a near-duplicate of a tuning case sits in the held-out half and the held-out number quietly stops being held out.

A held-out set also wears out. Every time you look at it, it becomes a little more like a set you are tuning against, and after enough looks the two halves say the same thing. Look at it on a schedule, before a release rather than on every change, and record how many times you have. The eval-driven development topic puts numbers on how fast this happens; the short version is that a set you consult twenty times has already given up several points of honesty.

Version it#

If the set changes, scores across time are not comparable. Version it, and when you add cases, note that the baseline moved. Otherwise you will one day celebrate an improvement that was actually a change of dataset.

The procedure is two lines of discipline. When you add cases, run the current system on the new version of the set and record that number as the new baseline, before anyone changes anything. And when you compare across a set change, compare on the cases both versions share. The eval statistics section works through what the alternative looks like: the same system, unchanged, appearing to improve by two points because sixty easy cases arrived in March.

One more rule, learned expensively. Never delete a case because it fails. A case that has failed for three months is not a bad case, it is the only honest record you have of a problem nobody prioritised. Mark it as known-failing if you must, keep it in the set, and let the number carry the debt.

You have a set of cases. The next problem is scoring the output, which for open-ended text has no obvious answer. This section works through the two tools available: plain code, which is free and cannot itself be wrong, and a model grading against a written rubric, which can be. Most of the section is about the second, because a judge nobody has checked against their own labels produces a number with no meaning attached to it.

What the case asks for                       Grader            Cost per case

one right string, from a fixed set           exact match       nothing
one right string, free-form wording          exact + normalise nothing
a number, a date, an id                      parse and compare nothing
valid structure: JSON, fields, types         schema check      nothing
an internal rule: totals, ranges, citations  constraint check  nothing
generated code                               run the tests     a sandbox
the right documents used                     set overlap       nothing
wording that could be phrased many ways      judge             a model call
faithful, on-topic, right tone               judge             a model call
which of two answers is better               pairwise judge    two model calls

Check with code wherever you can#

Deterministic checks are fast, free and cannot themselves be wrong. Exhaust them before reaching for a model.

• Is it valid JSON matching the schema?
• Are the required fields present and the right type?
• Is the total correct, given the line items?
• Does every citation point at a document that exists?
• Is it under the length limit?
• Did the side effect happen: was the record written, the email sent?

A surprising amount of quality is checkable this way. Teams reach for a judge far earlier than they need to.

Exact match, and how far normalising gets you#

Exact match is the grader to want. It is free, it has no opinion, and it never drifts. It fails on trivia, which is what normalising is for: strip the whitespace, fold the case, drop trailing punctuation, and the same answer stops being six different answers.

import re, unicodedata

def normalise(s):
    s = unicodedata.normalize("NFKC", s).strip().lower()
    s = s.replace("’", "'")                   # the curly apostrophe
    s = re.sub(r"[.!?;:]+$", "", s)           # trailing punctuation only
    s = re.sub(r"\s+", " ", s)
    return s

CASES = [
    ("refund window, annual plan", "14 days", "14 days"),
    ("refund window, monthly plan", "30 days", "30 days."),
    ("can I pause my plan", "no", "No"),
    ("support hours", "9am to 5pm", "9am  to 5pm"),
    ("price of the pro plan", "$1,200", "1200"),
    ("who signs the contract", "the account owner", "the owner of the account"),
]

exact = sum(exp == got for _, exp, got in CASES)
norm  = sum(normalise(exp) == normalise(got) for _, exp, got in CASES)
print("raw exact match  ", exact, "/", len(CASES))
print("after normalising", norm, "/", len(CASES))
for name, exp, got in CASES:
    if normalise(exp) != normalise(got):
        print("  still fails:", name, "|", exp, "->", got)

# raw exact match   1 / 6
# after normalising 4 / 6
#   still fails: price of the pro plan | $1,200 -> 1200
#   still fails: who signs the contract | the account owner -> the owner of the account

The two that remain are different problems and want different answers. The price is a number wearing formatting, so parse it and compare numerically; that is a grader, not a normaliser. The last is a genuine paraphrase, and no amount of string surgery will fix it. That is the boundary: normalise formatting, parse values, and hand real paraphrase to something else.

Fuzzy matching, and why the threshold does not exist#

The obvious next move is a similarity score with a threshold. It is worth doing once, on your own data, because the result is usually decisive in the other direction.

from difflib import SequenceMatcher

def ratio(a, b):
    """Character-level similarity, in the standard library."""
    return SequenceMatcher(None, a.lower(), b.lower()).ratio()

def token_f1(a, b):
    """Word overlap, the metric behind most question-answering scores."""
    ta, tb = a.lower().split(), b.lower().split()
    common = sum(min(ta.count(w), tb.count(w)) for w in set(ta))
    if common == 0:
        return 0.0
    p, r = common / len(tb), common / len(ta)
    return 2 * p * r / (p + r)

PAIRS = [
    ("the account owner", "the owner of the account"),
    ("refunds take 5 working days", "refunds are processed in five working days"),
    ("the refund is available", "the refund is not available"),
    ("cancel any time", "you can cancel whenever you like"),
]
print(f"{'ratio':>7}{'tokenF1':>9}  pair")
for a, b in PAIRS:
    print(f"{ratio(a, b):7.2f}{token_f1(a, b):9.2f}  {a!r} / {b!r}")

#   ratio  tokenF1  pair
#    0.54     0.75  'the account owner' / 'the owner of the account'
#    0.70     0.50  'refunds take 5 working days' / 'refunds are processed in five working days'
#    0.92     0.89  'the refund is available' / 'the refund is not available'
#    0.51     0.22  'cancel any time' / 'you can cancel whenever you like'

A single negation flips the meaning and changes almost no characters, and that is exactly the failure your users will report. Embedding similarity behaves the same way for the same reason. So fuzzy matching is not a grader for correctness. It is useful for two narrower jobs: flagging cases whose output changed a lot between two runs, so you know what to read, and deduplicating inputs as in the dataset section. Correctness needs either a parse or a judge.

Programmatic checks: structure, then rules#

Where the output has a shape, the grader is an ordinary function and it should report why it failed rather than false. Collect reasons, because the reason is what you will group and count later.

import json

SCHEMA = {
    "order_id": str,
    "items": list,
    "total_cents": int,
    "refundable": bool,
}

def check_structure(raw):
    """Returns a list of reasons the output is unusable. Empty means it passed."""
    problems = []
    try:
        obj = json.loads(raw)
    except json.JSONDecodeError as e:
        return ["not valid JSON: " + str(e)]
    if not isinstance(obj, dict):
        return ["top level is " + type(obj).__name__ + ", expected object"]
    for field, want in SCHEMA.items():
        if field not in obj:
            problems.append("missing field " + field)
        elif not isinstance(obj[field], want) or isinstance(obj[field], bool) != (want is bool):
            problems.append(field + " is " + type(obj[field]).__name__ + ", expected " + want.__name__)
    for extra in set(obj) - set(SCHEMA):
        problems.append("unexpected field " + extra)
    if not problems:
        line_total = sum(i.get("cents", 0) for i in obj["items"])
        if line_total != obj["total_cents"]:
            problems.append(f"total_cents {obj['total_cents']} but line items sum to {line_total}")
        if obj["total_cents"] < 0:
            problems.append("total_cents is negative")
    return problems

OUTPUTS = [
    '{"order_id": "A-101", "items": [{"cents": 900}, {"cents": 2000}], "total_cents": 2900, "refundable": true}',
    'Sure! Here is the JSON:\n{"order_id": "A-101", "items": [], "total_cents": 0, "refundable": true}',
    '{"order_id": "A-102", "items": [{"cents": 4900}], "total_cents": 4900, "refundable": "yes"}',
    '{"order_id": "A-103", "items": [{"cents": 1000}, {"cents": 1000}], "total_cents": 1000, "refundable": false}',
]
for raw in OUTPUTS:
    print(check_structure(raw) or "pass")

# pass
# ['not valid JSON: Expecting value: line 1 column 1 (char 0)']
# ['refundable is str, expected bool']
# ['total_cents 1000 but line items sum to 2000']

The last two rows are the interesting ones. A string 'yes' where a boolean belongs is a schema failure, and a total that disagrees with its own line items is a constraint failure. They are different bugs with different fixes, and a grader returning a bare false would have merged them.

When the output is code, the grader is the test suite. Run the generated function against cases you wrote, count how many pass, and keep the first failing input, because that is what tells you what kind of wrong it is.

TESTS = [
    ((["a", "b", "a"],), {"a": 2, "b": 1}),
    (([],), {}),
    ((["x"],), {"x": 1}),
    ((["a", "a", "a"],), {"a": 3}),
]

CANDIDATES = {
    "v1": "def tally(xs):\n    out = {}\n    for x in xs:\n        out[x] = out.get(x, 0) + 1\n    return out\n",
    "v2": "def tally(xs):\n    out = {}\n    for x in xs:\n        out[x] = 1\n    return out\n",
    "v3": "def tally(xs):\n    return {x: xs.count(x) for x in xs[1:]}\n",
    "v4": "def tally(xs)\n    return {}\n",
}

def run_tests(source, tests):
    ns = {}
    try:
        exec(source, ns)            # sandbox this properly before running model output
    except SyntaxError as e:
        return 0, len(tests), "SyntaxError: " + e.msg
    fn = ns.get("tally")
    if fn is None:
        return 0, len(tests), "no function named tally"
    passed, first_bad = 0, ""
    for args, want in tests:
        try:
            got = fn(*args)
        except Exception as e:
            got = type(e).__name__
        if got == want:
            passed += 1
        elif not first_bad:
            first_bad = f"tally({args[0]!r}) -> {got!r}, wanted {want!r}"
    return passed, len(tests), first_bad

for name, src in CANDIDATES.items():
    passed, total, why = run_tests(src, TESTS)
    print(f"{name}  {passed}/{total}  {why}")

# v1  4/4
# v2  2/4  tally(['a', 'b', 'a']) -> {'a': 1, 'b': 1}, wanted {'a': 2, 'b': 1}
# v3  3/4  tally(['x']) -> {}, wanted {'x': 1}
# v4  0/4  SyntaxError: expected ':'

Two things this toy leaves out and production cannot. Run model-written code in a sandbox with no network and no filesystem, in a separate process, because you are executing text a model produced from an input a user controls. And put a timeout on it, since a plausible-looking loop that never terminates will otherwise hang the eval run rather than failing a case.

LLM as judge#

For things code cannot check, such as whether a summary is faithful or a tone is right, a model can grade against a written rubric. The rubric has to be specific enough that two people applying it would agree.

Weak rubric:   "Is this a good summary?"

Usable rubric:
  Check one thing: does the summary claim anything the source
  does not support? Ignore style and length.

  FAIL if any claim is missing from the source or contradicts it.
  PASS if every claim can be found in the source.

  One sentence of critique, then PASS or FAIL on its own line.

Validate the judge#

Everything below runs against a stub judge rather than a hosted one, so the protocol executes offline and the same numbers come out every time. The stub has three knobs: how often it flags a real failure, how often it flags a good output, and later how much the order or the length of an answer sways it. Those knobs are what you are trying to measure in a real judge, and the code around them is exactly what you would run against one.

import numpy as np
rng = np.random.default_rng(7)

N = 400
human_fail = rng.random(N) < 0.12       # your own labels: 12% of outputs are bad

def stub_judge(human_fail, catch, false_alarm, rng):
    """Stands in for a hosted judge so this runs offline. catch is how often it
    flags a real failure, false_alarm how often it flags a good output."""
    r = rng.random(len(human_fail))
    return np.where(human_fail, r < catch, r < false_alarm)

naive = stub_judge(human_fail, catch=0.45, false_alarm=0.03, rng=rng)

human = ["FAIL" if x else "PASS" for x in human_fail]
judge = ["FAIL" if x else "PASS" for x in naive]
print(human.count("FAIL"), "failures in", len(human), "labelled outputs")

# 44 failures in 400 labelled outputs

fails  = [j for h, j in zip(human, judge) if h == "FAIL"]
passes = [j for h, j in zip(human, judge) if h == "PASS"]
print(f"caught {fails.count('FAIL')}/{len(fails)} failures, "
      f"accepted {passes.count('PASS')}/{len(passes)} passes")

# caught 20/44 failures, accepted 350/356 passes

Quote those numbers whenever you quote the score. A single agreement figure can hide a judge that passes everything, because when failures are rare, passing everything still agrees most of the time. The next block shows how large that hiding effect is, and what to report instead.

Raw agreement is the share of outputs where you and the judge said the same thing. Cohen's kappa is the same quantity after subtracting the agreement two graders would reach by chance alone, given how often each of them says fail. It is defined as the observed agreement minus the chance agreement, divided by the room left above chance, so 1 means perfect and 0 means no better than guessing at the right rate.

def agreement(a, b):
    return float((a == b).mean())

def kappa(a, b):
    """Agreement above what two graders would reach by chance."""
    po = agreement(a, b)
    pa1, pb1 = a.mean(), b.mean()
    pe = pa1 * pb1 + (1 - pa1) * (1 - pb1)
    return float("nan") if pe == 1 else float((po - pe) / (1 - pe))

print("raw agreement", round(agreement(human_fail, naive), 3))
print("kappa        ", round(kappa(human_fail, naive), 3))

lazy = np.zeros(N, dtype=bool)          # a judge that passes everything
print("passes-everything judge: agreement", round(agreement(human_fail, lazy), 3),
      " kappa", round(kappa(human_fail, lazy), 3))

from sklearn.metrics import cohen_kappa_score
print("sklearn check", round(cohen_kappa_score(human_fail, naive), 3))

print()
print("base rate   agreement of a judge that passes everything")
for base in (0.02, 0.05, 0.12, 0.30):
    print(f"{base:>8.0%}   {1 - base:>5.0%}")

# raw agreement 0.925
# kappa         0.533
# passes-everything judge: agreement 0.89  kappa 0.0
# sklearn check 0.533
#
# base rate   agreement of a judge that passes everything
#       2%     98%
#       5%     95%
#      12%     88%
#      30%     70%

That table is the reason to bother. On a good system, where failures are the two per cent you are hunting for, raw agreement is pinned near 98% no matter how bad the judge is, and the last two points of it carry all the information. Kappa moves over the whole range instead. The rough reading most people use is that below 0.4 the judge is not usable, 0.4 to 0.6 is weak, 0.6 to 0.8 is usable for tracking a number over time, and above 0.8 is as good as a second human. Those bands are convention rather than law, so quote the number itself.

Now fix the rubric: name the failure mode precisely, say what to ignore, and give one worked example of a pass and one of a fail. In the stub that is a judge that catches 85% of failures instead of 45%. Watch what each metric says about the improvement.

rubric = stub_judge(human_fail, catch=0.85, false_alarm=0.04, rng=rng)

print("naive judge : agreement", round(agreement(human_fail, naive), 3),
      " kappa", round(kappa(human_fail, naive), 3))
print("with rubric : agreement", round(agreement(human_fail, rubric), 3),
      " kappa", round(kappa(human_fail, rubric), 3))
fails_n = int(human_fail.sum())
print(f"caught {int((rubric & human_fail).sum())}/{fails_n} real failures, "
      f"false alarms on {int((rubric & ~human_fail).sum())}/{int((~human_fail).sum())} good outputs")

# naive judge : agreement 0.925  kappa 0.533
# with rubric : agreement 0.948  kappa 0.763
# caught 40/44 real failures, false alarms on 17/356 good outputs

Then the question everyone asks next: how many labels does this take? Bootstrap the kappa you just measured, by resampling your labelled set with replacement and recomputing, to see how much of it is the judge and how much is the sample.

def kappa_interval(human, judge, n_labels, boot=2000, seed=1):
    r = np.random.default_rng(seed)
    ks = []
    for _ in range(boot):
        idx = r.integers(0, n_labels, n_labels)
        ks.append(kappa(human[:n_labels][idx], judge[:n_labels][idx]))
    ks = np.array(ks)                   # nan where a resample drew no failures
    lo, hi = np.nanpercentile(ks, [2.5, 97.5])
    return lo, hi

for n in (50, 100, 400):
    lo, hi = kappa_interval(human_fail, rubric, n)
    print(f"{n:>4} labels ({int(human_fail[:n].sum()):>2} of them failures): "
          f"kappa {kappa(human_fail[:n], rubric[:n]):.2f}, 95% interval {lo:.2f} to {hi:.2f}")

#   50 labels ( 6 of them failures): kappa 0.81, 95% interval 0.39 to 1.00
#  100 labels (11 of them failures): kappa 0.67, 95% interval 0.43 to 0.86
#  400 labels (44 of them failures): kappa 0.76, 95% interval 0.66 to 0.85

That is the practical rule. Count failures, not labels, and if your system is good enough that failures are rare, go and find them deliberately rather than labelling four thousand outputs to stumble on forty. Cases the judge and a code check disagree about are a good place to look, and so is anything a user complained about.

Measuring position bias#

When a judge compares two answers, which one is shown first changes the verdict. This is not a rumour to design around, it is a quantity you can measure in an afternoon: run every comparison twice with the order swapped and look at the gap between the two win rates. Here B is genuinely better, winning 58% of the time on merit, and the judge has a pull towards whichever answer it sees first.

import numpy as np
rng = np.random.default_rng(11)

def pairwise_stub(n, p_b_on_merit, position_pull, rng):
    """Each pair judged in both orders. Returns whether B won when it was
    shown second, and whether it won when it was shown first."""
    b_second = rng.random(n) < (p_b_on_merit - position_pull)
    b_first  = rng.random(n) < (p_b_on_merit + position_pull)
    return b_second, b_first

N = 600
b_second, b_first = pairwise_stub(N, p_b_on_merit=0.58, position_pull=0.15, rng=rng)

print("B wins when shown second:", f"{b_second.mean():.1%}")
print("B wins when shown first :", f"{b_first.mean():.1%}")
print("position gap            :", f"{b_first.mean() - b_second.mean():.1%}")
print("swap and average        :", f"{(b_first.mean() + b_second.mean()) / 2:.1%}")

both = (b_first & b_second).mean()
neither = (~b_first & ~b_second).mean()
print(f"consistent verdicts: B twice {both:.1%}, A twice {neither:.1%}, "
      f"flipped {1 - both - neither:.1%}")

# B wins when shown second: 44.5%
# B wins when shown first : 75.5%
# position gap            : 31.0%
# swap and average        : 60.0%
# consistent verdicts: B twice 32.8%, A twice 12.8%, flipped 54.3%

Three things follow. Always run both orders; it doubles the cost of a pairwise eval and is the cheapest correctness fix in this section. Report the gap between the two orders as a property of your judge, because it tells you how much to trust any comparison you cannot afford to run twice. And do not use the flip rate as your bias measure: over half the pairs here flip, but most of that is the judge's own inconsistency rather than position, which is why the gap between the two order-specific win rates is the number to quote.

The same swap trick answers a question nobody asks and everybody should: how often does the judge agree with itself? Run identical comparisons twice in the same order. Whatever fraction disagrees is a ceiling on how much any measurement from this judge can be trusted.

Measuring verbosity bias#

The other bias with a clean experiment behind it. Take answers you have already judged, append a paragraph that adds no information, and judge them again. The content is identical by construction, so any change in the pass rate is the judge responding to length.

import numpy as np

PADDING = (" It is worth noting that policies of this kind are designed with "
           "the customer in mind, and we are always happy to help further.")

def scoring_stub(answers, quality, length_weight, rng):
    """Pass or fail on one answer at a time. length_weight is how much the
    judge is swayed by how long the answer is."""
    words = np.array([len(a.split()) for a in answers])
    pull = length_weight * (words - 25) / 25
    return (quality + pull + rng.normal(0, 0.15, len(answers))) > 0.5

base = ["The annual plan can be refunded within 14 days of purchase, "
        "and the refund reaches your card in about 5 working days."] * 300
quality = np.random.default_rng(3).normal(0.55, 0.2, 300)   # same answers, same quality

for weight, label in ((0.25, "judge as written"), (0.02, "after 'ignore length'")):
    plain  = scoring_stub(base, quality, weight, np.random.default_rng(5))
    padded = scoring_stub([a + PADDING for a in base], quality, weight,
                          np.random.default_rng(5))
    print(f"{label:<24} plain {plain.mean():>5.0%}   padded {padded.mean():>5.0%}   "
          f"gap {padded.mean() - plain.mean():>+5.0%}")
print("words:", len(base[0].split()), "plain,", len((base[0] + PADDING).split()), "padded")

# judge as written         plain   52%   padded   87%   gap  +35%
# after 'ignore length'    plain   56%   padded   61%   gap   +4%
# words: 22 plain, 46 padded

A judge with that bias does more than inflate a number. It will reliably prefer the more verbose of two prompts, so if you are hill-climbing against it you will be taught to make your product more long-winded, one accepted change at a time, while the score goes up. The same probe with the padding replaced by confident phrasing measures a related bias, and the one with a citation appended to an unsupported claim measures a third.

Judge biases to control for#

• Length: judges prefer longer answers. Say in the rubric that length is not a factor.
• Self-preference: a model tends to prefer output written in its own style. Consider a different model as judge.
• Position: when comparing two answers, the order affects the verdict. Run both orders and average.
• Leniency: judges pass too much by default. Give a concrete example of a failure in the rubric.

Every one of those mitigations is a hypothesis until you re-measure. The loop is the same each time: probe for the bias, change one thing in the rubric, run the probe again, then recompute kappa against your labels to make sure the fix did not cost you accuracy elsewhere. A rubric line that removes verbosity bias by making the judge reject everything has not helped, and only the kappa will tell you.

Two more habits are worth the trouble. Keep the labelled set split, tuning the rubric on one half and reporting kappa on the other, for the same reason the dataset section splits the eval set. And pin the judge: its model version, its rubric text and its temperature are part of your measuring instrument, so changing any of them invalidates comparisons with last month exactly as changing the case set does.

Writing the rubric itself, including what to put in it and what a good critique-then-verdict prompt looks like, is covered in the LLM-as-judge topic. This section is the measurement around it: what the number means, how far it is from the truth, and which of the biases you have actually got.

An eval that has to be run by hand gets run when someone already suspects a problem, which is the one situation where it tells you least. This section is about wiring it into the ordinary flow of changes so it runs whether or not anyone is worried, and about what it should report: cost and latency beside quality, since changes routinely trade one against the others. It also covers keeping the set alive by feeding real failures back into it.

Automate it or it will not happen#

An eval you run manually gets run when you remember, which is when you already suspect a problem. The value is in catching the regression you did not suspect, and that only happens if it runs on every change.

# in CI, on every pull request
python -m evals.run --set core --compare-to main

# fails the build if the score drops more than 2 points

The arithmetic decides the design, so do it once with your own numbers rather than arguing about it. Cases, cost per case, seconds per case, how much you run in parallel, and how many pull request runs a day the team actually produces.

CASES_FULL, CASES_SMOKE = 240, 40
COST_PER_CASE = 0.006          # input + output tokens for one case, in dollars
SECONDS_PER_CASE = 2.4
CONCURRENCY = 8
PR_RUNS_PER_DAY = 45

def run_cost(n):
    return n * COST_PER_CASE, n * SECONDS_PER_CASE / CONCURRENCY

for name, n, per_day in (("smoke, every push", CASES_SMOKE, PR_RUNS_PER_DAY),
                         ("full, nightly", CASES_FULL, 1),
                         ("full, every push", CASES_FULL, PR_RUNS_PER_DAY)):
    cost, secs = run_cost(n)
    print(f"{name:<20} {cost:>6.2f} usd  {secs:>5.0f}s per run   "
          f"{cost * per_day * 30:>8.2f} usd per month")

# smoke, every push      0.24 usd     12s per run     324.00 usd per month
# full, nightly          1.44 usd     72s per run      43.20 usd per month
# full, every push       1.44 usd     72s per run    1944.00 usd per month

What goes in the smoke tier is a real decision and not a random sample. Forty cases chosen at random are mostly cases that have passed every day for a year and will pass again, which is forty cases of nothing. Choose instead by what has ever moved.

• Every case that has changed verdict in the last three months. These are the cases near the edge of what the system can do, and they move first.
• One case per failure mode from your error analysis, so a resurrected bug is caught by name.
• The cheapest case in each category, so no category is silently unrepresented.
• The two or three cases that cover anything expensive or irreversible: money moved, data deleted, an email sent.
• Everything that is a code check rather than a judge call, since those are free and fast. There is no reason to sample them at all.

The nightly tier runs everything, against the full set and with more runs per case, and it is where the number you quote comes from. The pull request tier exists only to stop obviously broken changes from merging, and it should be understood by everyone as a smoke alarm rather than a measurement.

Score is not the only output#

Report cost and latency alongside quality, every time. Changes frequently trade one for another, and a quality-only report hides that completely.

                 before    after
pass rate        84%       86%      better
mean cost        $0.031    $0.074   2.4x worse
p95 latency      2.1s      5.8s     worse

Better, if you can afford it. That is a decision, not a detail.

The gate: deciding what fails the build#

A gate has two failure modes and they pull in opposite directions. Too tight and it fires on clean changes, the team learns to re-run it until it passes, and within a month it is advisory. Too loose and a real regression merges. Both are measurable before you choose, by simulating the gate against a system that has not changed and against one that has.

import numpy as np
from math import comb

def exact_paired_p(down, up):
    """Sign test on the cases that changed verdict, in either direction."""
    n, k = down + up, max(down, up)
    if n == 0:
        return 1.0
    tail = sum(comb(n, i) for i in range(k, n + 1)) / 2 ** n
    return min(1.0, 2 * tail)

N = 120
BASE = 0.86

def one_run(true_rate, flip, rng):
    """Baseline and candidate on the same N cases. flip is the share of cases
    whose output is not stable, so they get re-rolled at the new true rate."""
    a = rng.random(N) < BASE
    b = a.copy()
    move = rng.random(N) < flip
    b[move] = rng.random(move.sum()) < true_rate
    return a, b

def gate_rates(true_rate, flip, trials=3000, seed=0):
    rng = np.random.default_rng(seed)
    fires = np.zeros(3)
    for _ in range(trials):
        a, b = one_run(true_rate, flip, rng)
        drop = a.mean() - b.mean()
        down = int((a & ~b).sum())
        up = int((~a & b).sum())
        fires += [drop > 0,
                  drop > 0.05,
                  exact_paired_p(down, up) < 0.05 and down > up]
    return fires / trials

print("            any drop   > 5 points   paired test")
no_change = gate_rates(BASE, 0.25)
print(f"no change   {no_change[0]:>8.0%} {no_change[1]:>11.0%} {no_change[2]:>12.0%}   <- false alarms")
for true_rate, label in ((0.78, "8 pt worse"), (0.66, "20 pt worse")):
    r = gate_rates(true_rate, 1.0)
    print(f"{label:<12}{r[0]:>8.0%} {r[1]:>11.0%} {r[2]:>12.0%}   <- caught")

#             any drop   > 5 points   paired test
# no change        42%          2%           1%   <- false alarms
# 8 pt worse       94%         74%          31%   <- caught
# 20 pt worse     100%        100%          95%   <- caught

Read the columns as three different products. Any-drop catches everything and cries wolf 42% of the time. A five-point threshold is the compromise that works at this size: it fires on 2% of clean runs and catches three quarters of an eight-point regression and all of a large one. The paired test is the most rigorous and the least useful here, because at 120 cases it demands more evidence than a moderate regression provides, and a gate that misses two thirds of real eight-point regressions is not doing its job.

So the honest reading is that at 120 cases you cannot have both. If an eight-point regression must be caught reliably, the answer is not a cleverer test, it is a bigger nightly set. Set the pull request threshold where the false alarms are rare, accept that it only catches the large breaks, and let the nightly run on a larger set with a paired comparison be the thing that catches the rest.

When the eval itself is flaky#

Before blaming a change, find out what the suite does when nothing changes at all. Run it ten times against the same commit. The spread you see is the floor on every comparison you will ever make with it, and it is usually larger than the effects people are arguing about.

import numpy as np
rng = np.random.default_rng(12)

N, RUNS = 120, 10
stable = rng.random(N)
# Each case has its own chance of passing. Most sit near 0 or 1; a few in the
# middle, and those are the ones that decide the score from run to run.
p = np.where(stable < 0.70, 0.99, np.where(stable < 0.88, 0.02, rng.uniform(0.2, 0.8, N)))

results = rng.random((RUNS, N)) < p          # RUNS x N pass/fail, system unchanged
scores = results.mean(axis=1)
print("ten runs of an unchanged system:", " ".join(f"{s:.3f}" for s in scores))
print(f"spread {100 * (scores.max() - scores.min()):.1f} points")

passes = results.sum(axis=0)
minority = np.minimum(passes, RUNS - passes)  # how often the case disagreed with itself
flaky = minority >= 2
print(f"{int(flaky.sum())} of {N} cases gave both verdicts at least twice in ten runs")

stable_only = results[:, ~flaky].mean(axis=1)
print(f"on the {int((~flaky).sum())} stable cases: spread "
      f"{100 * (stable_only.max() - stable_only.min()):.1f} points")

# ten runs of an unchanged system: 0.717 0.733 0.758 0.692 0.767 0.733 0.733 0.742 0.750 0.750
# spread 7.5 points
# 19 of 120 cases gave both verdicts at least twice in ten runs
# on the 101 stable cases: spread 4.0 points

What to do with those nineteen cases depends on why they flip, and the answer is always to read them rather than to delete them. A case is flaky for one of three reasons, and only one of them is the case's fault.

• The grader is ambiguous. Two outputs that a person would call equally good land on opposite sides of it. Fix the grader; this is the most common cause and the one worth the time.
• The system genuinely fails this input half the time. The case is telling the truth and the product is the problem. Move it out of the gate and into a reliability report, which is the same distinction the agent evaluation section draws.
• The judge is inconsistent with itself. Measure that directly by judging the same output twice, and if it is high, the rubric needs work before anything else is worth doing.

Quarantine is the holding pen, not the fix. Keep quarantined cases running and reported, with their flip rate visible, and keep them out of the gate until someone has decided which of the three they are. A quarantine list that never shrinks is a team that has stopped doing error analysis.

Making a non-deterministic suite behave#

Several small habits remove noise that has nothing to do with the system under test, and they are worth applying before you start enlarging sets.

• Set temperature to zero for graded runs. It does not make output identical on a hosted model, but it removes the largest source of variation for free.
• Fix the case order and any sampling seed, so a run is reproducible from its recorded configuration. Store the seed with the result.
• Pin the model version explicitly rather than taking the moving alias, and record it in the run. A silent model update looks exactly like a regression, and this is how you tell them apart.
• Cache by the hash of prompt, model and parameters. Unchanged cases then cost nothing, which is what makes running the suite on every push affordable.
• Run every case the same number of times and average. Three runs per case costs three times as much and cuts the standard error of the score by nearly half.
• Keep the judge's temperature, rubric and version in the same pinned configuration as the model. It is part of the instrument.

Keep the history#

Store every run. Direction over months tells you more than any single score, and it is what lets you answer whether the system is improving or just changing.

Store enough per run that a surprise is explainable six weeks later: the commit, the model version, the set version, the judge version, the seed, the per-case verdicts and the cost. The per-case verdicts are the ones people leave out and then need, because the useful question after a drop is never what is the score, it is which cases changed. Answering that from stored results takes a second; rerunning last month's system to find out takes an afternoon and often is not possible at all, because the model version is gone.

Feed production back in#

Sample real traffic into the eval set as the product evolves, and add every genuine failure as a case. Over time the set stops being something you assembled and becomes a description of what the system is actually asked to do.

Make it a rule with a step attached, in the way a good bug process does. Any incident, any complaint, any thumbs-down that turns out to be real, becomes a case in the set before the fix merges, and the case goes in failing. A fix that cannot be expressed as a case that was red and is now green is a fix nobody can defend in three months, and it is the single habit that most reliably turns an eval suite from a thing that was built into a thing that is used.

Evaluation gets harder as soon as a system takes more than one step. There is no single correct transcript to compare against, the same task can be done well by several different routes, and a mistake at step two may only become visible at step nine. This section is about what to check instead, which is mostly the state of the world at the end, and about tracing backwards to the step where a run first went wrong. Repeated runs and step budgets belong here too, because an agent that succeeds one time in three is not working.

Why agents need a different approach#

A single call has one input and one output. An agent makes a sequence of decisions, and two runs on the same task can take completely different routes. Judging the final message alone misses most of what matters, and judging every step against one expected path punishes an agent for finding a different valid route.

Check the outcome in the world#

Where the agent changes something, grade the change itself. If it was asked to cancel Jack's second order and issue a refund, look at the database afterwards. Is that order cancelled? Is the other order untouched? Is there exactly one refund of the right amount? This is more reliable than asking a judge whether the transcript sounds right.

def grade(env_before, env_after):
    orders = env_after.orders_for("jack")
    return all([
        orders["A-102"].status == "cancelled",
        orders["A-101"].status == env_before.orders_for("jack")["A-101"].status,
        len(env_after.refunds_for("A-102")) == 1,
        env_after.refunds_for("A-102")[0].amount == 49.00,
    ])

Notice what the second line is doing. Checking that the target changed is the easy half; checking that nothing else did is the half that catches the expensive failures. An agent that cancels the right order and also cancels the other one passes any grader that only looks at what it was asked about.

This only works if each task starts from a known world and you can compare against it, which means a sandbox you can reset. Build it as a fixture: seed the database, run the task, diff the whole state, assert on the diff. Then a grader is the list of changes you expected, and everything else in the diff is a finding. Agents that touch the real world need the same thing with fakes at the boundary, because an eval suite that sends real emails will be run once.

Then look at the path#

Outcome checks tell you whether it worked. They do not tell you why it failed. For failing runs, walk the trace and mark the first step that went wrong. Over many runs, tally which step type fails first most often.

First failing step, 40 failed runs
understand request     6
choose tool           11
fill tool arguments   15
interpret tool result  5
final reply            3

A table like this turns 'the agent is flaky' into 'most failures start when the agent fills in tool arguments'. That is fixable, often with a better tool description, a stricter argument schema, or a lookup tool so it stops guessing ids.

Most of that table can be produced by code rather than by reading. Walk the steps in order, apply the checks that do not need to know the task, and stop at the first one that trips. The checks below are generic enough to be worth writing once: a tool that does not exist, an argument of the wrong type, acting on an id the agent never looked up, and retrying an identical call that has already failed.

from collections import Counter

ALLOWED = {"lookup_order", "cancel_order", "issue_refund", "send_email"}
ARG_TYPES = {"order_id": str, "amount_cents": int, "reason": str}

def first_bad_step(trace):
    """Walk the steps and return (index, what went wrong), or None if the path
    is clean. Only the first is reported: later steps inherit the mistake."""
    seen_ids = set()
    for i, step in enumerate(trace):
        if step["tool"] not in ALLOWED:
            return i, "tool does not exist"
        for key, value in step["args"].items():
            if key not in ARG_TYPES:
                return i, "unknown argument " + key
            if not isinstance(value, ARG_TYPES[key]):
                return i, "argument " + key + " has the wrong type"
        oid = step["args"].get("order_id")
        if step["tool"] != "lookup_order" and oid is not None and oid not in seen_ids:
            return i, "acted on an id it never looked up"
        nxt = trace[i + 1] if i + 1 < len(trace) else None
        if step["result"] == "not found" and nxt is not None:
            if nxt["tool"] == step["tool"] and nxt["args"] == step["args"]:
                return i + 1, "retried the identical failing call"
        if step["tool"] == "lookup_order" and step["result"] != "not found":
            seen_ids.add(oid)
    return None

TRACES = [
    [{"tool": "lookup_order", "args": {"order_id": "A-101"}, "result": "ok"},
     {"tool": "cancel_order", "args": {"order_id": "A-101"}, "result": "ok"}],
    [{"tool": "cancel_order", "args": {"order_id": "A-102"}, "result": "ok"}],
    [{"tool": "lookup_order", "args": {"order_id": "A-103"}, "result": "not found"},
     {"tool": "lookup_order", "args": {"order_id": "A-103"}, "result": "not found"}],
    [{"tool": "lookup_order", "args": {"order_id": "A-104"}, "result": "ok"},
     {"tool": "issue_refund", "args": {"order_id": "A-104", "amount_cents": "4900"},
      "result": "ok"}],
    [{"tool": "refund_order", "args": {"order_id": "A-105"}, "result": "error"}],
]

tally = Counter()
for t in TRACES:
    bad = first_bad_step(t)
    print(bad if bad else "clean")
    if bad:
        tally[bad[1]] += 1
print(tally.most_common())

# clean
# (0, 'acted on an id it never looked up')
# (1, 'retried the identical failing call')
# (1, 'argument amount_cents has the wrong type')
# (0, 'tool does not exist')
# [('acted on an id it never looked up', 1), ('retried the identical failing call', 1),
#  ('argument amount_cents has the wrong type', 1), ('tool does not exist', 1)]

Run that over every failing trace and the tally is your first-failing-step table, built automatically and updated on every eval run. What it cannot judge is whether the agent chose the right tool for this particular request, which still needs either an expected-tool annotation on the task or a person reading. Automate the checks that do not need the task, and spend the reading time on the ones that do.

Tool calls are checkable#

• Did it call a tool at all when one was needed, and not call one when none was needed?
• Was it the right tool?
• Do the arguments pass the schema, and are they the right values for this request?
• Did it retry sensibly after an error, or loop on the same failing call?

Most of these are code checks against the trace. Save judges for questions like whether the reply correctly explained what happened.

Run it more than once#

An agent that succeeds on a task half the time is not half useful. For a customer who hits it three times, it fails at least once almost nine times in ten. Run each task several times and report two numbers: the share of runs that succeed, and the share of tasks where every run succeeds. The second is what users experience as reliability.

# Ten tasks, four runs each. 1 means the end state was correct.
results = {
    "cancel one of two orders":      [1, 1, 1, 1],
    "refund a partial amount":       [1, 0, 1, 1],
    "change the delivery address":   [1, 1, 1, 1],
    "apply a discount code":         [1, 1, 0, 1],
    "split an order in two":         [0, 1, 0, 1],
    "close the account":             [1, 1, 1, 1],
    "reissue an invoice":            [1, 1, 1, 0],
    "move a subscription to annual": [1, 1, 1, 1],
    "merge two accounts":            [0, 0, 1, 0],
    "extend a trial":                [1, 1, 1, 1],
}
k = 4
avg_success = sum(sum(r) for r in results.values()) / (k * len(results))
all_k_pass  = sum(all(r) for r in results.values()) / len(results)
print(f"per run {avg_success:.0%}   all {k} runs {all_k_pass:.0%}")
for task, runs in results.items():
    if not all(runs) and any(runs):
        print(f"  intermittent: {task} {runs}")

# per run 80%   all 4 runs 50%
#   intermittent: refund a partial amount [1, 0, 1, 1]
#   intermittent: apply a discount code [1, 1, 0, 1]
#   intermittent: split an order in two [0, 1, 0, 1]
#   intermittent: reissue an invoice [1, 1, 1, 0]
#   intermittent: merge two accounts [0, 0, 1, 0]

Four runs is few enough that a task passing all four is weaker evidence than it feels. The left table below is what to expect from a task of a given real reliability; the right one is why agents with many steps are hard, since a run only succeeds if every step does.

print("chance a task passes every one of k runs, by its real per-run rate")
print(f"{'real rate':>10}" + "".join(f"{'k=' + str(k):>8}" for k in (1, 4, 10)))
for p in (0.99, 0.95, 0.90, 0.80, 0.70):
    print(f"{p:>10.0%}" + "".join(f"{p ** k:>8.0%}" for k in (1, 4, 10)))

print()
print("a run of s steps, each independently right with probability q")
print(f"{'q':>6}" + "".join(f"{'s=' + str(s):>8}" for s in (5, 10, 20, 40)))
for q in (0.99, 0.98, 0.95, 0.90):
    print(f"{q:>6}" + "".join(f"{q ** s:>8.0%}" for s in (5, 10, 20, 40)))

# chance a task passes every one of k runs, by its real per-run rate
#  real rate     k=1     k=4    k=10
#        99%     99%     96%     90%
#        95%     95%     81%     60%
#        90%     90%     66%     35%
#        80%     80%     41%     11%
#        70%     70%     24%      3%
#
# a run of s steps, each independently right with probability q
#      q     s=5    s=10    s=20    s=40
#   0.99     95%     90%     82%     67%
#   0.98     90%     82%     67%     45%
#   0.95     77%     60%     36%     13%
#    0.9     59%     35%     12%      1%

The right-hand table also says where to spend effort. Going from 95% to 98% per step more than doubles the success rate of a twenty-step task. Adding a recovery path, so a failed step is retried with the error fed back rather than ending the run, changes the exponent rather than the base and is usually the larger win. Both are measurable with the same harness.

Budgets are part of correctness#

An agent that solves the task in forty steps and two dollars may be a failure in production. Record steps, tokens, cost and wall time per run, and set limits. A run that exceeds the budget counts as a failure, whatever the final answer was.

import numpy as np
rng = np.random.default_rng(21)

RUNS = 200
steps = rng.integers(4, 9, RUNS)
stuck = rng.random(RUNS) < 0.12              # runs that thrash before finishing
steps = np.where(stuck, steps + rng.integers(10, 30, RUNS), steps)
cost = steps * 0.004 + rng.normal(0, 0.0015, RUNS).clip(0)
correct = rng.random(RUNS) < 0.88

BUDGET_STEPS, BUDGET_COST = 15, 0.08
over = (steps > BUDGET_STEPS) | (cost > BUDGET_COST)

print(f"outcome correct             {correct.mean():.0%}")
print(f"within step and cost budget {(~over).mean():.0%}")
print(f"both                        {(correct & ~over).mean():.0%}")
print(f"median steps {int(np.median(steps))}, p95 steps {int(np.percentile(steps, 95))}")
print(f"median cost {np.median(cost):.3f}, p95 cost {np.percentile(cost, 95):.3f}, "
      f"mean cost {cost.mean():.3f}")
print(f"the {over.mean():.0%} of runs over budget spend "
      f"{100 * cost[over].sum() / cost.sum():.0f}% of the money")

# outcome correct             90%
# within step and cost budget 88%
# both                        78%
# median steps 7, p95 steps 24
# median cost 0.028, p95 cost 0.098, mean cost 0.035
# the 12% of runs over budget spend 36% of the money

Quote the p95, never the mean, for steps and latency. The mean hides exactly the runs that hurt: here half the money goes on an eighth of the runs, and those are also the ones a user waits through. Setting the step limit at the p95 of successful runs is a reasonable starting point, and then every run that hits the limit is either a loop to fix or a task that genuinely needs a bigger budget.

Eval scores move about on their own, and without a sense of how much, you cannot tell an improvement from a coincidence. This section is the minimum statistics for the job: putting an error bar on a pass rate, comparing two versions on the same cases so the comparison is tighter, allowing for the model's own run-to-run variation, and working out how many cases you need before a number means anything. It is the difference between measuring a change and believing in one.

The problem#

Version A passes 40 of 50 cases. Version B passes 42. Is B better? Almost certainly you cannot tell. Teams ship changes like this every week and call them improvements. Knowing how much a score can wobble by chance, and so whether a difference is statistically significant, is what separates measurement from wishful thinking.

A pass rate has an error bar#

A pass rate from a sample is an estimate. With few cases the range of true values consistent with it is wide. The Wilson interval is a good default for pass rates, and it behaves sensibly near 0% and 100% where the simple formula does not.

from math import sqrt

def wilson(passed, n, z=1.96):
    p = passed / n
    denom = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / denom
    half = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / denom
    return centre - half, centre + half

print(wilson(40, 50))    # about (0.67, 0.89)
print(wilson(400, 500))  # about (0.76, 0.83)

At fifty cases, 80% could plausibly be anything from the high sixties to the high eighties. Ten times the cases narrows that to a few points either side.

Compare on the same cases#

When both versions run on the same set, most cases pass or fail for both. Only the cases where they disagree carry information about which is better. Count them.

                 B passes   B fails
A passes            38          2
A fails              4          6

Disagreements: B won 4, A won 2.

Six disagreements split four to two is the kind of result you get from a coin tossed six times. A simple test on the disagreements alone, known as McNemar's test, formalises this. With so few disagreements it will say what intuition should: no evidence that B is better.

from math import comb

def paired_p(b_wins, a_wins):
    """Two-sided exact test on the disagreeing cases only."""
    n, k = b_wins + a_wins, max(b_wins, a_wins)
    tail = sum(comb(n, i) for i in range(k, n + 1)) / 2 ** n
    return min(1.0, 2 * tail)

print(paired_p(4, 2))   # about 0.69, no evidence either way
print(paired_p(18, 4))  # about 0.004, real difference

One number is not a result#

Everything above treats the eval as a single pass rate, which is how it gets reported and how the most expensive mistakes get through. A set covers several categories, a change rarely affects them equally, and the aggregate is a weighted average that can rise while the part you care about falls. Print the breakdown every time, with an interval per category, and read that before the headline.

# (category, cases, passed before, passed after)
RESULTS = [
    ("track an order", 120, 108, 114),
    ("change a plan",   60,  49,  53),
    ("invoice query",   40,  30,  35),
    ("account access",  30,  23,  25),
    ("refund request",  50,  40,  32),
]

print(f"{'category':<16}{'n':>4}{'before':>8}{'after':>7}{'change':>8}   95% intervals")
tot_n = tot_b = tot_a = 0
for name, n, before, after in RESULTS:
    tot_n, tot_b, tot_a = tot_n + n, tot_b + before, tot_a + after
    lo_b, hi_b = wilson(before, n)
    lo_a, hi_a = wilson(after, n)
    print(f"{name:<16}{n:>4}{before / n:>8.0%}{after / n:>7.0%}{(after - before) / n:>+8.0%}"
          f"   before {lo_b:.0%}-{hi_b:.0%}  after {lo_a:.0%}-{hi_a:.0%}")
print(f"{'aggregate':<16}{tot_n:>4}{tot_b / tot_n:>8.0%}{tot_a / tot_n:>7.0%}"
      f"{(tot_a - tot_b) / tot_n:>+8.0%}")

# category           n  before  after  change   95% intervals
# track an order   120     90%    95%     +5%   before 83%-94%  after 90%-98%
# change a plan     60     82%    88%     +7%   before 70%-89%  after 78%-94%
# invoice query     40     75%    88%    +12%   before 60%-86%  after 74%-95%
# account access    30     77%    83%     +7%   before 59%-88%  after 66%-93%
# refund request    50     80%    64%    -16%   before 67%-89%  after 50%-76%
# aggregate        300     83%    86%     +3%

This is not a rare pathology. It is what you should expect whenever a change helps the common path, since the common path dominates the average by construction. A prompt that adds a summarising instruction improves four categories of question and breaks the one where the detail mattered, and the aggregate says yes.

Two working rules follow. Report per category always, and make the gate look at categories rather than the total, so a large drop in any one of them fails the build even when the headline rose. And accept what the intervals in that table are saying: a thirty-case category carries a range of roughly fifteen points either side, so per-category numbers need per-category sample sizes. If a category matters enough to gate on, it needs enough cases to measure, which in practice means at least thirty and preferably more.

The set moved, not the system#

The other way an aggregate lies is when the denominator changes underneath it. Cases get added every month, which is correct practice, and each addition shifts the mix. Then two numbers that were never comparable get compared in a slide.

# Same system, same behaviour on every case it already had. All that changed
# is that 60 cases from an easy category were added to the set in March.
feb = {"n": 300, "passed": 249}
mar = {"n": 360, "passed": 249 + 57}
print(f"February  {feb['passed']}/{feb['n']} = {feb['passed'] / feb['n']:.0%}")
print(f"March     {mar['passed']}/{mar['n']} = {mar['passed'] / mar['n']:.0%}")
print(f"reported improvement {100 * (mar['passed'] / mar['n'] - feb['passed'] / feb['n']):+.1f} points")
print(f"on the 300 cases both months share: {feb['passed'] / feb['n']:.0%} then {249 / 300:.0%}")

# February  249/300 = 83%
# March     306/360 = 85%
# reported improvement +2.0 points
# on the 300 cases both months share: 83% then 83%

So version the set, re-baseline when it changes, and when you plot a score over months, plot it on the cases that existed for the whole period rather than on whatever the set contained that week. The same discipline, applied to a set you tune against repeatedly rather than one you add to, is what the eval-driven development topic measures under improving against a number: twenty worthless changes against a hundred cases produce twelve points of improvement that is not there.

The model varies too#

Run the same version twice and the score changes, because sampling and hosted inference are not perfectly repeatable. Before comparing versions, run one version three to five times and look at the spread. If it moves by three points between identical runs, a two point improvement is noise.

How many cases you need#

• To catch large regressions, such as a feature breaking, twenty to fifty cases is enough.
• To detect a change of around five points reliably, you need several hundred cases, or a paired comparison with enough disagreements.
• Small improvements of one or two points need thousands of cases. Most teams cannot measure these, and should not claim them.

Those figures are worked out with a simulation in the product judgement topic, under choosing and changing models, where a genuine four-and-a-half point improvement is called correctly by fifty examples only one time in five. It is the same arithmetic in a different setting, and running it once on your own numbers is the fastest way to stop arguing about whether a two-point move is real.

One habit undoes all of this, and it is the one every team drifts into: running the eval, seeing no difference, changing something small, running it again, and stopping when a run finally looks good. Each look is another chance for noise to produce a winner, so the false-positive rate is not the 5% you think you are working to. The experiment design topic counts exactly this, and the numbers are worth carrying here: checking once gives 4.9% false positives, checking fifty times gives 32.1%. Decide the sample size and the criterion before the run, and let the run finish.