Data quality & labelling

Every evaluation you run compares an output against something you decided was correct, and that decision came from somewhere. This section is about where: writing guidelines people can actually apply, measuring whether two annotators agree, and reading the disagreements instead of averaging them away. When three people label the same hundred items differently the cause is almost never carelessness, it is a task definition with a hole in it. Find those holes here and every number you measure afterwards means more.

Labels are opinions until they agree#

Ask three people to label the same hundred items and they will not match. That gap is not incompetence. It is almost always ambiguity in the task definition that nobody noticed until it had to be applied.

Guidelines that work#

• Define each label with a positive and a negative example, not an adjective.
• Say explicitly what to do with ambiguous cases, because they will arrive.
• Include a not-applicable option, or people will force a label and pollute the data.
• Write them, have someone else label with them, then fix what they got wrong. The first draft is always incomplete.

Measure agreement, and not with a percentage#

The obvious measure is the share of items two people labelled the same way. Compute it, because it is the number everyone will ask for, and then do not report it on its own, because on any task where one answer is rare it is close to meaningless.

import numpy as np
from sklearn.metrics import cohen_kappa_score

rng = np.random.default_rng(5)
N = 1000

def pair(prevalence, skill):
    """Two annotators. 'skill' is the chance each applies the true label;
    otherwise they guess at the prevalence."""
    truth = rng.random(N) < prevalence
    def label():
        follow = rng.random(N) < skill
        return np.where(follow, truth, rng.random(N) < prevalence).astype(int)
    return label(), label()

for title, skill in [("two careful annotators", 0.85), ("two annotators with no skill at all", 0.0)]:
    print(title)
    print(f"{'prevalence':>11} {'raw agreement':>14} {'Cohen kappa':>12}")
    for prev in [0.50, 0.30, 0.10, 0.05, 0.02]:
        a, b = pair(prev, skill)
        print(f"{prev:>11.2f} {(a == b).mean():>14.3f} {cohen_kappa_score(a, b):>12.3f}")
    print()

two careful annotators
 prevalence  raw agreement  Cohen kappa
       0.50          0.873        0.745
       0.30          0.884        0.723
       0.10          0.954        0.716
       0.05          0.963        0.641
       0.02          0.987        0.780

two annotators with no skill at all
 prevalence  raw agreement  Cohen kappa
       0.50          0.500       -0.000
       0.30          0.574       -0.024
       0.10          0.803       -0.026
       0.05          0.913        0.019
       0.02          0.950        0.013

The second table is the one that should change how you talk about this. Those two annotators have no skill whatsoever: each is flipping a weighted coin, ignoring the item entirely. On a task where two per cent of items are positive they agree ninety-five per cent of the time, because they both say no to almost everything and two people saying no agree. A team reporting ninety-five per cent inter-annotator agreement on a rare-class task has reported nothing at all, and the number will be believed because it sounds high.

Cohen's kappa is the correction: it subtracts the agreement you would expect from the marginal rates alone and rescales, so that chance agreement is zero and perfect agreement is one. The no-skill rows sit between minus 0.03 and 0.02, which is what zero looks like with a thousand items. The careful annotators sit between 0.64 and 0.78 across every prevalence, which is the honest picture of a genuinely somewhat-noisy task, and notice that raw agreement swung from 0.873 to 0.987 across the same rows while the underlying quality did not move at all.

• Report kappa, with raw agreement beside it and the class prevalence beside that. Kappa alone hides that a low score on a balanced task may be fine.
• Rough reading: below 0.2 the task is not defined; 0.2 to 0.4 is poor; 0.4 to 0.6 is workable for a hard subjective task; above 0.6 is good; above 0.8 is either an easy task or a leaked answer.
• For ordered labels such as a one-to-five rating, use weighted kappa, so being one apart counts as a smaller disagreement than being four apart.
• For more than two annotators, Krippendorff's alpha does the same job and tolerates missing labels, which a real annotation queue will have.

Read the disagreements#

Disagreements are the most informative part of the data. They point directly at the cases your definition does not cover, and resolving them improves every future label. Doing this properly is a loop rather than a report, and it converges faster than people expect.

1. 1.Pull every item the two annotators labelled differently. On fifty items with kappa around 0.6 this is typically ten to fifteen items, which is an hour of work rather than a project.
2. 2.Sort them into buckets by why they disagree, not by what they are about. The usual buckets are a genuinely ambiguous case, a term in the guideline that two people read differently, a case the guideline never contemplated, and an actual slip.
3. 3.Only the last bucket is an annotator problem. Every other bucket is a guideline change, and the guideline change is what you are here for.
4. 4.Rewrite the guideline with the disputed cases as worked examples, with the decision and the reason. A rule with a hard example attached is applied consistently; a rule stated as a principle is not.
5. 5.Re-label a fresh sample and measure kappa again. If it has not moved, the rewrite addressed the wrong thing, which is usually a sign the disagreement was genuine ambiguity and the label set needs a category rather than a clarification.

The outcome people do not expect is that sometimes the right answer is to change the label set rather than the wording. If two careful readers keep splitting on the same kind of item, that kind of item is a third thing, and forcing it into one of two boxes is what is producing the noise. Adding a category, or an explicit not-applicable, usually recovers more agreement than any amount of clarifying prose.

How many labels, and how many twice#

Two budget questions come up immediately and both have arithmetic answers rather than opinions. The first is how many items to label at all, and for an eval set it is set by the precision you need on the score. A proportion measured on n items has a standard error of at most half the square root of one over n, so 100 items gives plus or minus about 10 points at 95 per cent confidence, 400 gives 5 points, and 1,000 gives about 3. An eval set of 50 items cannot distinguish a model that scores 80 from one that scores 68, which is most of the comparisons anyone wants to make.

items    worst-case 95% interval on a proportion
   50     +/- 14 points
  100     +/- 10 points
  400     +/-  5 points
1,000     +/-  3 points
4,000     +/-  1.5 points

The second is how many items to label twice. Double-labelling everything doubles the cost and is almost never the right call. Double-label a sample, large enough to measure kappa with a usable interval — a hundred to two hundred items is the usual answer — and single-label the rest. The sample is a measurement of the process, not a quality gate on individual items, and it should be repeated on each new batch rather than done once at the start, because agreement decays as annotators join and the guideline is forgotten.

Same rules for eval sets#

An eval set is a labelled set, and the golden set everything else is measured against. Everything above applies: write down what a good answer means before looking at outputs, have someone else apply your criteria, and treat your disagreements as evidence the criteria were vague.

One thing is different, and it is worth saying because it catches people who have done this well for training data. A training set tolerates noise: a model trained on labels that are five per cent wrong is usually a little worse, and the errors average out across thousands of examples. An eval set does not average anything out, because you are reading the number directly. Five per cent label noise in an eval set puts a five-point floor under every comparison you make with it, and no amount of running the eval more times reduces that floor. Spend the labelling budget on the eval set first.

What to skip#

• Skip raw agreement as a headline number. It rises with class imbalance and says nothing about whether anyone read the item.
• Skip buying more labels when kappa is low. More labels at the same disagreement rate is more noise at greater cost; fix the definition first.
• Skip adjudicating disagreements by majority vote without reading them. The vote produces a label and throws away the finding, which was the ambiguity.
• Worth memorising: kappa over raw agreement, that agreement caps model performance, and the square-root rule for how big an eval set has to be.

Label error rates of several percent are normal, including in datasets everyone trusts, and a duplicate straddling your train and test split turns a test score into a memory test. This section finds both: out-of-fold predictions to rank examples by how confidently the model disagrees with their label, hashing and fuzzy matching for duplicates, nearest-neighbour search between train and test for overlap. It ends with schema checks that stop a broken extract before anything trains on it, and slicing the evaluation so a group the model is bad at cannot hide inside a good average.

Use the model to find wrong labels#

Train with cross-validation so every example gets a prediction from a model that did not train on it. Examples where the model confidently disagrees with the label are candidates for label errors. Review the most confident disagreements by hand. The cleanlab library automates a principled version of this idea.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import cross_val_predict

rng = np.random.default_rng(2)
X, y_true = make_classification(n_samples=10000, n_features=20, n_informative=10,
                                class_sep=1.2, flip_y=0.0, random_state=0)
flipped = rng.choice(len(y_true), 300, replace=False)      # 3% of labels are wrong
y = y_true.copy()
y[flipped] = 1 - y[flipped]
is_wrong = np.zeros(len(y), dtype=bool)
is_wrong[flipped] = True

model = HistGradientBoostingClassifier(max_iter=200, random_state=0)
probs = cross_val_predict(model, X, y, cv=5, method="predict_proba")
confidence_in_given_label = probs[range(len(y)), y]
order = confidence_in_given_label.argsort()               # review from the top

print(f"{'review the top':>15} {'planted errors found':>21} {'precision':>10} {'recall':>8}")
for k in [50, 100, 300, 600, 1000]:
    found = is_wrong[order[:k]].sum()
    print(f"{k:>15} {found:>21} {found / k:>10.1%} {found / 300:>8.1%}")
print("300 random rows would have contained about",
      int(round(300 * is_wrong.mean())), "errors")

 review the top  planted errors found  precision   recall
             50                    50     100.0%    16.7%
            100                   100     100.0%    33.3%
            300                   251      83.7%    83.7%
            600                   294      49.0%    98.0%
           1000                   296      29.6%    98.7%
300 random rows would have contained about 9 errors

That last line is the number to carry into an argument about review budget. Three hundred rows of annotator time spent on randomly chosen examples surfaces about nine errors; the same three hundred rows spent on the top of this ranking surfaces 251. It is roughly a factor of thirty on the same human cost, and the only machinery is out-of-fold predictions you were already able to produce.

The shape of the precision column tells you where to stop. It is perfect for the first hundred, still 84 per cent at three hundred, and falls to 49 per cent by six hundred — at which point half of what a reviewer looks at is correctly labelled and merely difficult. Review down the list until precision drops to where the reviewer's time is better spent elsewhere, and do not expect to reach the last few errors: recall at 98 per cent costs twice the review of recall at 84 per cent.

Duplicates#

Exact duplicates are easy to drop with a hash. Near duplicates, such as the same document with a different footer, need fuzzy matching: MinHash for text at scale, embedding similarity for meaning. Duplicates inflate some examples' weight and, worse, leak between splits.

Train and test overlap#

If near-copies of test examples are in the training set, test scores measure memory. Check by searching the training set for the nearest neighbour of every test example and reading the closest pairs. This is the same contamination problem that affects public LLM benchmarks.

Stop bad data at the door#

Everything above finds problems in data you already have. This is the other half: a suite that runs on each arriving extract and refuses to let a broken one through. Libraries such as pandera or Great Expectations will express it declaratively, and the declarative form is easier to read than to learn from, so here it is written out. Six kinds of check, and the reason to name them separately is that each catches a different family of upstream mistake.

schema        the columns exist and hold the kind of thing they should
range         values sit inside bounds the business guarantees
nullity       the share of missing values has not moved
cardinality   categorical columns contain only values you have seen
freshness     the newest row is recent enough to be today's data
volume        the row count is close to the recent baseline

import numpy as np, pandas as pd
from pandas.api import types as pdt

rng = np.random.default_rng(11)

def extract(n=20000, day="2026-03-01", rng=rng):
    """One day's orders, as the warehouse hands them over."""
    df = pd.DataFrame({
        "order_id": np.arange(n),
        "amount": rng.lognormal(3.4, 0.7, n).round(2),
        "country": rng.choice(["GB", "DE", "FR", "ES", "IT"], n, p=[.4, .2, .2, .1, .1]),
        "signup_days": rng.integers(0, 3000, n).astype(float),
        "ts": pd.Timestamp(day) - pd.to_timedelta(rng.integers(0, 20 * 3600, n), unit="s"),
    })
    df.loc[rng.random(n) < 0.02, "signup_days"] = np.nan
    return df

def corrupt(df, rng=rng):
    """What an upstream release did overnight: country lower-cased, and one
    service started sending amount in pence. No exception anywhere."""
    out = df.copy()
    out["country"] = out["country"].str.lower()
    pence = rng.random(len(out)) < 0.15
    out.loc[pence, "amount"] = out.loc[pence, "amount"] * 100
    return out, int(pence.sum())

KINDS = {"order_id": pdt.is_integer_dtype, "amount": pdt.is_float_dtype,
         "country": pdt.is_string_dtype, "signup_days": pdt.is_float_dtype,
         "ts": pdt.is_datetime64_any_dtype}
COUNTRIES = {"GB", "DE", "FR", "ES", "IT"}

def check(df, baseline_rows, now):
    """Six checks, each returning a (kind, message) pair when it fails."""
    fails = []
    for col, ok in KINDS.items():
        if col not in df.columns:
            fails.append(("schema", f"{col} missing"))
        elif not ok(df[col]):
            fails.append(("schema", f"{col} is {df[col].dtype}, wrong kind"))
    outside = int((~df["amount"].between(0, 10_000)).sum())
    if outside:
        fails.append(("range", f"amount outside [0, 10000] on {outside} rows"))
    nulls = df["signup_days"].isna().mean()
    if nulls > 0.05:
        fails.append(("nullity", f"signup_days {nulls:.1%} null, limit 5%"))
    unseen = sorted(set(df["country"].unique()) - COUNTRIES)
    if unseen:
        fails.append(("cardinality", f"country has unseen values {unseen[:5]}"))
    lag = (now - df["ts"].max()).total_seconds() / 3600
    if lag > 6:
        fails.append(("freshness", f"newest row is {lag:.1f}h old, limit 6h"))
    ratio = len(df) / baseline_rows
    if not 0.8 <= ratio <= 1.2:
        fails.append(("volume", f"{len(df)} rows, {ratio:.2f}x the 7-day median"))
    return fails

clean = extract()
broken, n_corrupt = corrupt(clean)
now = pd.Timestamp("2026-03-01 02:00")
for name, df in [("clean", clean), ("broken", broken)]:
    res = check(df, baseline_rows=20000, now=now)
    print(f"{name}: {'PASS' if not res else str(len(res)) + ' failures'}")
    for kind, msg in res:
        print(f"   {kind:<12} {msg}")

clean: PASS
broken: 2 failures
   range        amount outside [0, 10000] on 135 rows
   cardinality  country has unseen values ['de', 'es', 'fr', 'gb', 'it']

Now count what it missed. The corruption multiplied 3,015 rows by a hundred, and the range check caught 135 of them — four and a half per cent. The other 2,880 are amounts between about 300 and 10,000, which are entirely plausible orders, and they sailed through every check in the suite. The cardinality check did its job loudly on the country column, and it only did so because the change there produced values that had never been seen before. Had the upstream release lower-cased nothing and only changed the units, the suite would have printed PASS on an extract in which one row in seven was wrong by a factor of a hundred.

Why silent corruption is the risk that matters#

A pipeline that crashes wakes someone up in an hour. A pipeline that quietly produces plausible nonsense trains a model, passes its evaluation on the same corrupted split, ships, and is discovered by a business metric drifting three months later. The cost is not the incident, it is the three months and everything built on top during them. Given the choice, prefer the loud failure every time: a schema that rejects an unexpected column is doing you a favour.

The way to catch the quiet version is to add a seventh check, on the shape of the distribution rather than on individual values. Summary statistics on the same corrupted column make the case for which statistic to use.

def psi(ref, cur, bins=10):
    edges = np.quantile(ref, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    r = np.clip(np.histogram(ref, edges)[0] / len(ref), 1e-6, None)
    c = np.clip(np.histogram(cur, edges)[0] / len(cur), 1e-6, None)
    return float(np.sum((c - r) * np.log(c / r)))

print("rows corrupted:", n_corrupt)
for stat, fn in [("median", lambda s: s.median()), ("mean", lambda s: s.mean()),
                 ("p99", lambda s: s.quantile(0.99))]:
    print(f"{stat:>7}  clean {fn(clean['amount']):>9.2f}   broken {fn(broken['amount']):>9.2f}")
print("PSI on amount:", round(psi(clean["amount"], broken["amount"]), 3))

rows corrupted: 3015
 median  clean     30.17   broken     35.00
   mean  clean     38.29   broken    622.66
    p99  clean    155.61   broken   8838.10
PSI on amount: 0.137

Which statistic you monitor decides whether you see this. The median is robust, which is exactly why it is useless here: it barely moves, because the corruption is a minority of rows. The mean and the upper percentiles are sensitive, which is normally a complaint about them and is the property you want in a monitor. PSI at 0.137 lands in the investigate-this band. So monitor a robust statistic to understand the typical case and a sensitive one to catch corruption, and never only the median.

What to alert on and what to merely record#

A suite like the one above produces a few dozen numbers per table per day, and a real pipeline has thirty tables. Alerting on all of it produces a channel nobody reads. The split that works is by what the receiver can do about it right now.

• Block the pipeline, and page: schema violations, freshness, and volume outside the baseline. These mean the data is wrong rather than surprising, and training or serving on it is worse than being late.
• Warn, and let the run continue: nullity and range failures on a small share of rows, unseen categories under a threshold. Somebody should look today, and stopping everything is a worse trade.
• Record only: every statistic in the suite, every day, for every column, whether it passed or not. This is the history that turns an investigation from an argument into a chart, and it costs a row in a table.
• Never alert on something with no owner and no action. An alert that arrives where nobody can act on it trains everyone to ignore the channel, including for the alerts that do have an action.

The recorded half is the part people skip and the part that pays off, because it is what tells you whether today's number is unusual. A nullity of 3 per cent means nothing on its own; 3 per cent against a 0.2 per cent history for ninety days is the whole finding. Store the statistics for everything and alert on almost none of it.

Once the suite is a page long, express it declaratively rather than as a function. Pandera is the smallest option and reads as a schema.

import pandera.pandas as pa

schema = pa.DataFrameSchema({
    "amount":   pa.Column(float, pa.Check.ge(0)),
    "country":  pa.Column(str, pa.Check.isin(sorted(COUNTRIES))),
    "label":    pa.Column(int, pa.Check.isin([0, 1])),
})
frame = pd.DataFrame({"amount": [12.0, 4.5], "country": ["GB", "DE"], "label": [0, 1]})
print(schema.validate(frame).shape)          # (2, 3): passes, returns the frame

try:
    schema.validate(pd.DataFrame({"amount": [-1.0], "country": ["ZZ"], "label": [0]}))
except pa.errors.SchemaError as e:
    print("raised:", type(e).__name__)        # raised: SchemaError

Look at slices#

An overall score can hide a group where the model is bad: one country, one device type, new customers. Break the evaluation down by the slices that matter to the business. Poor slices often turn out to be data problems, such as few examples or inconsistent labels in that group.