Building the dataset
By the end: You have a task set that tells you the truth rather than what you want to hear.
- Sourcing from real traffic
- Covering the distribution, not just the easy path
- Train and test splits, and why they apply here
- Versioning, so a score means something over time
New to evaluation? An eval is a fixed set of test cases you run your system against, plus a rule for deciding whether each answer was good enough. It is a test suite for something that never gives the same answer twice: instead of checking the output matches a stored string, you check it satisfies what you asked for — the right fields present, the right document cited, the refund policy not invented. Why it matters
An eval is only as honest as the cases in it, and the natural way to build a set, sitting down and thinking up inputs, produces the one thing you do not want: a test your system passes. This section is about where cases actually come from, what has to be in the set for a score to mean anything, and how to keep that score comparable as the set grows. Get this wrong and you still get a number every week, it just will not track reality.
Where a case came from What it is good for How many
production failure proving one bug is fixed all of them
escalation or thumbs-down the failures users noticed all of them
random sample of traffic the overall number 100-500
stratified sample by intent per-category numbers 20+ per category
hand-written edge case empty, huge, hostile input 10-30
model-generated input filling a category you lack as few as possibleUse real inputs#
Invented test cases are unconsciously shaped to be answerable, so they measure a system handling the inputs you imagined rather than the ones it gets. Pull from logs.
- Common cases, in roughly the proportion they occur.
- Every failure you have already seen in production. These are the highest-value cases you own.
- Edge cases: empty input, enormous input, wrong language, adversarial input.
- Cases where the correct answer is to refuse or to say nothing was found.
The empty case is always missing
Almost nobody includes the input where the right answer is 'there is nothing here'. It is also the case models handle worst, because they are strongly inclined to produce something.
Sampling, or how the rare categories vanish#
Taking a hundred conversations at random from the log feels like the neutral choice and is not, because the categories that matter least by volume often matter most by consequence. Here is last month's traffic and what a random sample of a hundred actually contained.
import numpy as np
rng = np.random.default_rng(2)
# Share of last month's traffic, by what the user was asking for.
MIX = {
"track an order": 0.41,
"change a plan": 0.22,
"refund request": 0.16,
"invoice query": 0.11,
"account access": 0.07,
"cancel and complain": 0.03,
}
names = list(MIX)
weights = np.array(list(MIX.values()))
log = rng.choice(len(names), size=8000, p=weights)
def counts(sample):
return {names[i]: int((sample == i).sum()) for i in range(len(names))}
random_100 = rng.choice(log, size=100, replace=False)
print("random 100 from the log")
for k, v in counts(random_100).items():
print(f" {k:<22}{v:>4}")
# Stratified: at least 8 of everything, the rest in proportion.
floor = 8
extra = 100 - floor * len(names)
plan = {n: floor + int(round(extra * w)) for n, w in MIX.items()}
print("stratified 100, floor of", floor)
for k, v in plan.items():
print(f" {k:<22}{v:>4}")
print("total", sum(plan.values()))
# random 100 from the log stratified 100, floor of 8
# track an order 47 track an order 29
# change a plan 14 change a plan 19
# refund request 20 refund request 16
# invoice query 14 invoice query 14
# account access 3 account access 12
# cancel and complain 2 cancel and complain 10
# total 100A floor per category fixes that, at a price: the set no longer looks like traffic, so the headline number drifts away from what users experience. Weight the categories back to the real mix when you report it, and keep both figures. The gap between them is usually a few points and is entirely a property of how you sampled.
# A stratified set no longer looks like traffic, so weight the categories back.
passed = {"track an order": 28, "change a plan": 17, "refund request": 12,
"invoice query": 11, "account access": 8, "cancel and complain": 5}
flat = sum(passed.values()) / sum(plan.values())
weighted = sum(MIX[n] * passed[n] / plan[n] for n in names)
print(f"unweighted mean of the stratified set {flat:.1%}")
print(f"weighted back to the traffic mix {weighted:.1%}")
# unweighted mean of the stratified set 81.0%
# weighted back to the traffic mix 86.1%A production failure is worth more than an invented case#
This is the claim worth testing rather than repeating. Suppose a bug affects 6% of traffic and you write a fix that works four times in five. A random eval set contains that bug only in proportion, so most of its cases are irrelevant to the question you are asking. Cases pulled from reports of the bug itself are all relevant. The difference is not a matter of taste.
import numpy as np
from math import comb
rng = np.random.default_rng(4)
def exact_paired_p(fixed, broken):
"""Two-sided sign test on the cases that changed verdict."""
n, k = fixed + broken, max(fixed, broken)
if n == 0:
return 1.0
tail = sum(comb(n, i) for i in range(k, n + 1)) / 2 ** n
return min(1.0, 2 * tail)
def detects(n_cases, share_affected, fix_rate=0.80, break_rate=0.02, trials=4000):
"""How often does this set show the fix, at p < 0.05 on a paired test?"""
wins = 0
for _ in range(trials):
affected = rng.random(n_cases) < share_affected
fixed = int((affected & (rng.random(n_cases) < fix_rate)).sum())
broken = int((~affected & (rng.random(n_cases) < break_rate)).sum())
wins += exact_paired_p(fixed, broken) < 0.05 and fixed > broken
return wins / trials
print("a bug in 6% of traffic, a fix that works 4 times in 5")
for n in (100, 200, 500, 1000):
print(f" random set of {n:>5}: detected {detects(n, 0.06):.0%} of the time")
for n in (20, 40):
print(f" {n} cases pulled from reports of that bug: {detects(n, 1.0):.0%}")
# random set of 100: detected 9% of the time
# random set of 200: detected 27% of the time
# random set of 500: detected 67% of the time
# random set of 1000: detected 95% of the time
# 20 cases pulled from reports of that bug: 100%
# 40 cases pulled from reports of that bug: 100%It is also the argument for keeping the two sets apart. The targeted set answers did this fix work, and it will always say yes more loudly than reality, because every case in it was selected for being broken. The traffic-shaped set answers how much this matters overall, and it needs to be large. Report them separately and neither one lies to you.
Write the expected outcome first#
Decide what a good answer looks like before you run the system. If you look at the output first, you will write a criterion the current version happens to satisfy, and your eval will certify whatever you already have.
Which means a case is more than an input and a string. It carries where it came from, which category it belongs to, which grader decides it and what that grader is looking for. Write that down as a record from the first day, because retrofitting provenance onto four hundred cases is a week nobody has.
CASE = {
"id": "refund-annual-day20",
"input": "Can I get a refund on the annual plan after 20 days?",
"category": "refund request",
"source": "ticket 48211, escalated 2026-02-11", # never 'made up by me'
"grader": "contains_all",
"expect": {"all_of": ["14 days"], "none_of": ["30 days", "no refund"]},
"added": "2026-02-12",
"set": "core-v4",
}Growing it without filling it with copies#
Sampling straight from logs gives you the same question thirty times, because most traffic is the same question. Duplicates do not just waste money; they silently reweight the set, so an improvement on one popular phrasing looks like an improvement everywhere. Deduplicate before you add, on word overlap rather than string equality.
LOG = [
"where is my order",
"Where is my order?",
"where is my order!!",
"can you tell me where my order is",
"my order hasn't arrived, where is it",
"how do I change my plan",
"how do i change my plan to annual",
"I want to downgrade to the free tier",
"the invoice says 1200 but I paid 1000",
"why was I charged twice this month",
"cancel my account and delete my data",
]
def shingles(text, k=3):
"""Sets of k consecutive words. Cheap, no model, no embedding service."""
words = text.lower().replace("?", "").replace("!", "").replace(",", "").split()
if len(words) < k:
return {" ".join(words)}
return {" ".join(words[i:i + k]) for i in range(len(words) - k + 1)}
def jaccard(a, b):
return len(a & b) / len(a | b)
kept, dropped, sigs = [], [], []
for text in LOG:
s = shingles(text)
near = next((kept[i] for i, t in enumerate(sigs) if jaccard(s, t) >= 0.5), None)
if near is None:
kept.append(text)
sigs.append(s)
else:
dropped.append((text, near))
print(f"{len(LOG)} log lines -> {len(kept)} distinct cases")
for text, near in dropped:
print(f" dropped {text!r}")
print(f" near {near!r}")
# 11 log lines -> 8 distinct cases
# dropped 'Where is my order?'
# near 'where is my order'
# dropped 'where is my order!!'
# near 'where is my order'
# dropped 'how do i change my plan to annual'
# near 'how do I change my plan'Note what word overlap does not catch: it keeps the two order-tracking paraphrases that share almost no words, which is the right answer for a test set and the wrong answer if you were deduplicating for cost. Embeddings catch those too, and then start merging cases that differ only in the detail that matters. A category label maintained by hand beats both.
Split it#
This is borrowed from machine learning and applies exactly. If you tune prompts against the whole set, you overfit to it: the score climbs while real performance does not. Hold some cases back and never look at them while iterating.
train (70%) tune against these, look at them freely
test (30%) score only, never inspect while iterating
Report the test number. That is the one that predicts reality.Split by category rather than at random, so both halves contain every category, and split by user or conversation rather than by case where several cases came from the same conversation. Otherwise a near-duplicate of a tuning case sits in the held-out half and the held-out number quietly stops being held out.
A held-out set also wears out. Every time you look at it, it becomes a little more like a set you are tuning against, and after enough looks the two halves say the same thing. Look at it on a schedule, before a release rather than on every change, and record how many times you have. The eval-driven development topic puts numbers on how fast this happens; the short version is that a set you consult twenty times has already given up several points of honesty.
Version it#
If the set changes, scores across time are not comparable. Version it, and when you add cases, note that the baseline moved. Otherwise you will one day celebrate an improvement that was actually a change of dataset.
The procedure is two lines of discipline. When you add cases, run the current system on the new version of the set and record that number as the new baseline, before anyone changes anything. And when you compare across a set change, compare on the cases both versions share. The eval statistics section works through what the alternative looks like: the same system, unchanged, appearing to improve by two points because sixty easy cases arrived in March.
One more rule, learned expensively. Never delete a case because it fails. A case that has failed for three months is not a bad case, it is the only honest record you have of a problem nobody prioritised. Mark it as known-failing if you must, keep it in the set, and let the number carry the debt.
Where people go wrong
Adding cases to the set whenever something fails, without re-baselining. Scores then move for two reasons at once and you can no longer tell which.
If this comes up in an interview
Where do eval cases come from?
Real traffic, not imagination. Common cases in realistic proportions, every production failure you have seen, edge cases, and cases where the right answer is to refuse or return nothing.
Why split an eval set into train and test?
Tuning against every case overfits the prompt to that set, so the score improves while real performance does not. A held-out split you never inspect gives a number that still predicts reality.
How big should an eval set be?
Twenty real cases is enough to start and already catches regressions. A few hundred gives tighter confidence. Beyond that, variety matters far more than volume.
You have a bug that affects about 6% of traffic and a fix for it. How do you show the fix works?
Not with the general eval set. At 6% prevalence, a hundred random cases contain about six that exercise the bug, and a paired test on that few detects a working fix less than a tenth of the time. I would pull twenty to forty cases from the reports of that specific bug and use those to show it is fixed, then use the general set to check nothing else regressed and to say how much it mattered overall. Two sets, two questions.
Then they push further
Does the targeted set not overstate the improvement?
Yes, and deliberately. Every case in it was selected for being broken, so its pass rate cannot be read as a product metric. That is why it is reported separately from the traffic-shaped set rather than merged into it.
How do you sample the general set so rare categories are usable?
Stratify with a floor of twenty or thirty per category, then weight the categories back to the real traffic mix when reporting the headline number. Without the floor the rare categories land with two or three cases and their pass rate can only be 0, 50 or 100 per cent.
What gets marked up. Strong answers separate the question 'did this fix work' from 'how much does this matter', and use a different set for each. Weak answers propose adding a few cases to the existing set and re-running it.
Build this: Build a fifty-case set from real inputs, split it 70/30, and record where each case came from. Resist looking at the test split for a week.
Watch it explained
This is one section of Evaluation, 35 hours across 7 sections. Read it in full to get the whole topic in order.
Stop 30 of 50 on the AI Engineer route
0 of 50 topics done · 0h behind you, 603h to go
What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.