Where ground truth comes from

Every evaluation you run compares an output against something you decided was correct, and that decision came from somewhere. This section is about where: writing guidelines people can actually apply, measuring whether two annotators agree, and reading the disagreements instead of averaging them away. When three people label the same hundred items differently the cause is almost never carelessness, it is a task definition with a hole in it. Find those holes here and every number you measure afterwards means more.

Labels are opinions until they agree#

Ask three people to label the same hundred items and they will not match. That gap is not incompetence. It is almost always ambiguity in the task definition that nobody noticed until it had to be applied.

Guidelines that work#

• Define each label with a positive and a negative example, not an adjective.
• Say explicitly what to do with ambiguous cases, because they will arrive.
• Include a not-applicable option, or people will force a label and pollute the data.
• Write them, have someone else label with them, then fix what they got wrong. The first draft is always incomplete.

Measure agreement, and not with a percentage#

The obvious measure is the share of items two people labelled the same way. Compute it, because it is the number everyone will ask for, and then do not report it on its own, because on any task where one answer is rare it is close to meaningless.

import numpy as np
from sklearn.metrics import cohen_kappa_score

rng = np.random.default_rng(5)
N = 1000

def pair(prevalence, skill):
    """Two annotators. 'skill' is the chance each applies the true label;
    otherwise they guess at the prevalence."""
    truth = rng.random(N) < prevalence
    def label():
        follow = rng.random(N) < skill
        return np.where(follow, truth, rng.random(N) < prevalence).astype(int)
    return label(), label()

for title, skill in [("two careful annotators", 0.85), ("two annotators with no skill at all", 0.0)]:
    print(title)
    print(f"{'prevalence':>11} {'raw agreement':>14} {'Cohen kappa':>12}")
    for prev in [0.50, 0.30, 0.10, 0.05, 0.02]:
        a, b = pair(prev, skill)
        print(f"{prev:>11.2f} {(a == b).mean():>14.3f} {cohen_kappa_score(a, b):>12.3f}")
    print()

two careful annotators
 prevalence  raw agreement  Cohen kappa
       0.50          0.873        0.745
       0.30          0.884        0.723
       0.10          0.954        0.716
       0.05          0.963        0.641
       0.02          0.987        0.780

two annotators with no skill at all
 prevalence  raw agreement  Cohen kappa
       0.50          0.500       -0.000
       0.30          0.574       -0.024
       0.10          0.803       -0.026
       0.05          0.913        0.019
       0.02          0.950        0.013

The second table is the one that should change how you talk about this. Those two annotators have no skill whatsoever: each is flipping a weighted coin, ignoring the item entirely. On a task where two per cent of items are positive they agree ninety-five per cent of the time, because they both say no to almost everything and two people saying no agree. A team reporting ninety-five per cent inter-annotator agreement on a rare-class task has reported nothing at all, and the number will be believed because it sounds high.

Cohen's kappa is the correction: it subtracts the agreement you would expect from the marginal rates alone and rescales, so that chance agreement is zero and perfect agreement is one. The no-skill rows sit between minus 0.03 and 0.02, which is what zero looks like with a thousand items. The careful annotators sit between 0.64 and 0.78 across every prevalence, which is the honest picture of a genuinely somewhat-noisy task, and notice that raw agreement swung from 0.873 to 0.987 across the same rows while the underlying quality did not move at all.

• Report kappa, with raw agreement beside it and the class prevalence beside that. Kappa alone hides that a low score on a balanced task may be fine.
• Rough reading: below 0.2 the task is not defined; 0.2 to 0.4 is poor; 0.4 to 0.6 is workable for a hard subjective task; above 0.6 is good; above 0.8 is either an easy task or a leaked answer.
• For ordered labels such as a one-to-five rating, use weighted kappa, so being one apart counts as a smaller disagreement than being four apart.
• For more than two annotators, Krippendorff's alpha does the same job and tolerates missing labels, which a real annotation queue will have.

Read the disagreements#

Disagreements are the most informative part of the data. They point directly at the cases your definition does not cover, and resolving them improves every future label. Doing this properly is a loop rather than a report, and it converges faster than people expect.

1. 1.Pull every item the two annotators labelled differently. On fifty items with kappa around 0.6 this is typically ten to fifteen items, which is an hour of work rather than a project.
2. 2.Sort them into buckets by why they disagree, not by what they are about. The usual buckets are a genuinely ambiguous case, a term in the guideline that two people read differently, a case the guideline never contemplated, and an actual slip.
3. 3.Only the last bucket is an annotator problem. Every other bucket is a guideline change, and the guideline change is what you are here for.
4. 4.Rewrite the guideline with the disputed cases as worked examples, with the decision and the reason. A rule with a hard example attached is applied consistently; a rule stated as a principle is not.
5. 5.Re-label a fresh sample and measure kappa again. If it has not moved, the rewrite addressed the wrong thing, which is usually a sign the disagreement was genuine ambiguity and the label set needs a category rather than a clarification.

The outcome people do not expect is that sometimes the right answer is to change the label set rather than the wording. If two careful readers keep splitting on the same kind of item, that kind of item is a third thing, and forcing it into one of two boxes is what is producing the noise. Adding a category, or an explicit not-applicable, usually recovers more agreement than any amount of clarifying prose.

How many labels, and how many twice#

Two budget questions come up immediately and both have arithmetic answers rather than opinions. The first is how many items to label at all, and for an eval set it is set by the precision you need on the score. A proportion measured on n items has a standard error of at most half the square root of one over n, so 100 items gives plus or minus about 10 points at 95 per cent confidence, 400 gives 5 points, and 1,000 gives about 3. An eval set of 50 items cannot distinguish a model that scores 80 from one that scores 68, which is most of the comparisons anyone wants to make.

items    worst-case 95% interval on a proportion
   50     +/- 14 points
  100     +/- 10 points
  400     +/-  5 points
1,000     +/-  3 points
4,000     +/-  1.5 points

The second is how many items to label twice. Double-labelling everything doubles the cost and is almost never the right call. Double-label a sample, large enough to measure kappa with a usable interval — a hundred to two hundred items is the usual answer — and single-label the rest. The sample is a measurement of the process, not a quality gate on individual items, and it should be repeated on each new batch rather than done once at the start, because agreement decays as annotators join and the guideline is forgotten.

Same rules for eval sets#

An eval set is a labelled set, and the golden set everything else is measured against. Everything above applies: write down what a good answer means before looking at outputs, have someone else apply your criteria, and treat your disagreements as evidence the criteria were vague.

One thing is different, and it is worth saying because it catches people who have done this well for training data. A training set tolerates noise: a model trained on labels that are five per cent wrong is usually a little worse, and the errors average out across thousands of examples. An eval set does not average anything out, because you are reading the number directly. Five per cent label noise in an eval set puts a five-point floor under every comparison you make with it, and no amount of running the eval more times reduces that floor. Spend the labelling budget on the eval set first.

What to skip#

• Skip raw agreement as a headline number. It rises with class imbalance and says nothing about whether anyone read the item.
• Skip buying more labels when kappa is low. More labels at the same disagreement rate is more noise at greater cost; fix the definition first.
• Skip adjudicating disagreements by majority vote without reading them. The vote produces a label and throws away the finding, which was the ambiguity.
• Worth memorising: kappa over raw agreement, that agreement caps model performance, and the square-root rule for how big an eval set has to be.