Evaluation set sizes and statistics
How many cases you need before a difference is real, what a percentage point means at each size, and when to believe a judge.
What one number is worth
| Standard error of a rate | √(p(1−p) ÷ n) | The 95% interval is roughly ±2 standard errors. At p near 0.5 this is at its widest, which is the conservative case to plan with. |
|---|---|---|
| n = 100 | ±10 percentage points | A smoke test. It will catch a catastrophe and nothing subtler. Reporting 73% from 100 cases implies a precision you do not have. |
| n = 400 | ±5 percentage points | Enough to notice a large regression. Still not enough to decide between two candidate prompts differing by a few points. |
| n = 1,000 | ±3 percentage points | A reasonable working size for a decision-grade suite, if the cases are well chosen. |
| n = 2,500 | ±2 percentage points | About the smallest difference a hand-labelled offline suite can defend. Below this, label noise starts to matter as much as sample size. |
| n = 10,000 | ±1 percentage point | Rarely worth it for a hand-curated suite. At this size the labelling cost dominates, and label noise usually exceeds the precision gained. |
Comparing two systems
| Detecting a 10-point difference | about 400 cases per arm | At 80% power, 5% significance, rates near 50%. Rates further from 50% need fewer. |
|---|---|---|
| Detecting a 5-point difference | about 1,600 cases per arm | Quadrupling precision costs sixteen times the data. This is the single most useful thing to know before promising to detect a small improvement. |
| Detecting a 2-point difference | about 10,000 cases per arm | Which is usually the point at which the honest answer is that you cannot measure this offline, and the question belongs in an online experiment. |
| Paired testing | run both systems on the same cases and compare only where they disagree | McNemar's test. Removes the variance from case difficulty and often needs a third to a fifth of the cases an unpaired comparison would. Always do this when you can. |
| Multiple comparisons | 20 variants tested at p < 0.05 produce about one false winner by chance | Prompt sweeps are exactly this situation. Hold out a final confirmation set that the winner has never been scored on. |
| Rule of three | zero failures in n trials means failure rate is under about 3/n with 95% confidence | Zero out of 100 does not mean safe. It means the rate is plausibly as high as 3%. |
Judges and labels
| Cohen's kappa | below 0.4 poor · 0.4-0.6 moderate · 0.6-0.8 substantial · above 0.8 strong | Kappa corrects for agreement that would happen by chance, which raw percentage agreement does not. On a skewed rubric, 90% raw agreement can be a kappa near zero. |
|---|---|---|
| The ceiling on a judge | how well two careful humans agree with each other | Often 0.7-0.8 on subjective rubrics. A judge matching human-human agreement is as good as the task allows; demanding more than that is demanding the impossible. |
| Validating a judge | at least 100 human-labelled cases, stratified across the score range | Sampling only easy cases flatters the judge. Include the ambiguous ones deliberately, because those are the cases that decide whether a change shipped. |
| Pairwise versus absolute | pairwise preferences agree with humans substantially better than absolute scores | Ask which of two outputs is better rather than to rate one out of five. Randomise the order — position bias is real and measurable. |
| Revalidate on change | any change to the judge prompt or judge model invalidates prior scores | Version the judge alongside the system, and keep a frozen set so you can tell which of them moved. |
Building the set
| Minimum useful suite | about 100 cases drawn from real traffic | Better than 1,000 invented at a desk. Error analysis on real traces is where a good eval set comes from. |
|---|---|---|
| Stratify by failure mode | 20-50 cases per mode you have actually observed | A uniform random sample will underrepresent the rare failure that is the reason you are building this. |
| Hold out a confirmation set | 20-30% of cases, scored only when a change is about to ship | Iterating against every case you own is how a suite stops predicting production and becomes a training set for your prompts. |
| Refresh cadence | add new failures continuously; prune cases every system passes | A suite where everything passes measures nothing. The information is in the cases that still fail. |
Figures last checked 2026-09-13. Several of these age; check anything you are about to rely on.
Where these numbers come from