Finding and fixing bad data

Label error rates of several percent are normal, including in datasets everyone trusts, and a duplicate straddling your train and test split turns a test score into a memory test. This section finds both: out-of-fold predictions to rank examples by how confidently the model disagrees with their label, hashing and fuzzy matching for duplicates, nearest-neighbour search between train and test for overlap. It ends with schema checks that stop a broken extract before anything trains on it, and slicing the evaluation so a group the model is bad at cannot hide inside a good average.

Use the model to find wrong labels#

Train with cross-validation so every example gets a prediction from a model that did not train on it. Examples where the model confidently disagrees with the label are candidates for label errors. Review the most confident disagreements by hand. The cleanlab library automates a principled version of this idea.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import cross_val_predict

rng = np.random.default_rng(2)
X, y_true = make_classification(n_samples=10000, n_features=20, n_informative=10,
                                class_sep=1.2, flip_y=0.0, random_state=0)
flipped = rng.choice(len(y_true), 300, replace=False)      # 3% of labels are wrong
y = y_true.copy()
y[flipped] = 1 - y[flipped]
is_wrong = np.zeros(len(y), dtype=bool)
is_wrong[flipped] = True

model = HistGradientBoostingClassifier(max_iter=200, random_state=0)
probs = cross_val_predict(model, X, y, cv=5, method="predict_proba")
confidence_in_given_label = probs[range(len(y)), y]
order = confidence_in_given_label.argsort()               # review from the top

print(f"{'review the top':>15} {'planted errors found':>21} {'precision':>10} {'recall':>8}")
for k in [50, 100, 300, 600, 1000]:
    found = is_wrong[order[:k]].sum()
    print(f"{k:>15} {found:>21} {found / k:>10.1%} {found / 300:>8.1%}")
print("300 random rows would have contained about",
      int(round(300 * is_wrong.mean())), "errors")

 review the top  planted errors found  precision   recall
             50                    50     100.0%    16.7%
            100                   100     100.0%    33.3%
            300                   251      83.7%    83.7%
            600                   294      49.0%    98.0%
           1000                   296      29.6%    98.7%
300 random rows would have contained about 9 errors

That last line is the number to carry into an argument about review budget. Three hundred rows of annotator time spent on randomly chosen examples surfaces about nine errors; the same three hundred rows spent on the top of this ranking surfaces 251. It is roughly a factor of thirty on the same human cost, and the only machinery is out-of-fold predictions you were already able to produce.

The shape of the precision column tells you where to stop. It is perfect for the first hundred, still 84 per cent at three hundred, and falls to 49 per cent by six hundred — at which point half of what a reviewer looks at is correctly labelled and merely difficult. Review down the list until precision drops to where the reviewer's time is better spent elsewhere, and do not expect to reach the last few errors: recall at 98 per cent costs twice the review of recall at 84 per cent.

Duplicates#

Exact duplicates are easy to drop with a hash. Near duplicates, such as the same document with a different footer, need fuzzy matching: MinHash for text at scale, embedding similarity for meaning. Duplicates inflate some examples' weight and, worse, leak between splits.

Train and test overlap#

If near-copies of test examples are in the training set, test scores measure memory. Check by searching the training set for the nearest neighbour of every test example and reading the closest pairs. This is the same contamination problem that affects public LLM benchmarks.

Stop bad data at the door#

Everything above finds problems in data you already have. This is the other half: a suite that runs on each arriving extract and refuses to let a broken one through. Libraries such as pandera or Great Expectations will express it declaratively, and the declarative form is easier to read than to learn from, so here it is written out. Six kinds of check, and the reason to name them separately is that each catches a different family of upstream mistake.

schema        the columns exist and hold the kind of thing they should
range         values sit inside bounds the business guarantees
nullity       the share of missing values has not moved
cardinality   categorical columns contain only values you have seen
freshness     the newest row is recent enough to be today's data
volume        the row count is close to the recent baseline

import numpy as np, pandas as pd
from pandas.api import types as pdt

rng = np.random.default_rng(11)

def extract(n=20000, day="2026-03-01", rng=rng):
    """One day's orders, as the warehouse hands them over."""
    df = pd.DataFrame({
        "order_id": np.arange(n),
        "amount": rng.lognormal(3.4, 0.7, n).round(2),
        "country": rng.choice(["GB", "DE", "FR", "ES", "IT"], n, p=[.4, .2, .2, .1, .1]),
        "signup_days": rng.integers(0, 3000, n).astype(float),
        "ts": pd.Timestamp(day) - pd.to_timedelta(rng.integers(0, 20 * 3600, n), unit="s"),
    })
    df.loc[rng.random(n) < 0.02, "signup_days"] = np.nan
    return df

def corrupt(df, rng=rng):
    """What an upstream release did overnight: country lower-cased, and one
    service started sending amount in pence. No exception anywhere."""
    out = df.copy()
    out["country"] = out["country"].str.lower()
    pence = rng.random(len(out)) < 0.15
    out.loc[pence, "amount"] = out.loc[pence, "amount"] * 100
    return out, int(pence.sum())

KINDS = {"order_id": pdt.is_integer_dtype, "amount": pdt.is_float_dtype,
         "country": pdt.is_string_dtype, "signup_days": pdt.is_float_dtype,
         "ts": pdt.is_datetime64_any_dtype}
COUNTRIES = {"GB", "DE", "FR", "ES", "IT"}

def check(df, baseline_rows, now):
    """Six checks, each returning a (kind, message) pair when it fails."""
    fails = []
    for col, ok in KINDS.items():
        if col not in df.columns:
            fails.append(("schema", f"{col} missing"))
        elif not ok(df[col]):
            fails.append(("schema", f"{col} is {df[col].dtype}, wrong kind"))
    outside = int((~df["amount"].between(0, 10_000)).sum())
    if outside:
        fails.append(("range", f"amount outside [0, 10000] on {outside} rows"))
    nulls = df["signup_days"].isna().mean()
    if nulls > 0.05:
        fails.append(("nullity", f"signup_days {nulls:.1%} null, limit 5%"))
    unseen = sorted(set(df["country"].unique()) - COUNTRIES)
    if unseen:
        fails.append(("cardinality", f"country has unseen values {unseen[:5]}"))
    lag = (now - df["ts"].max()).total_seconds() / 3600
    if lag > 6:
        fails.append(("freshness", f"newest row is {lag:.1f}h old, limit 6h"))
    ratio = len(df) / baseline_rows
    if not 0.8 <= ratio <= 1.2:
        fails.append(("volume", f"{len(df)} rows, {ratio:.2f}x the 7-day median"))
    return fails

clean = extract()
broken, n_corrupt = corrupt(clean)
now = pd.Timestamp("2026-03-01 02:00")
for name, df in [("clean", clean), ("broken", broken)]:
    res = check(df, baseline_rows=20000, now=now)
    print(f"{name}: {'PASS' if not res else str(len(res)) + ' failures'}")
    for kind, msg in res:
        print(f"   {kind:<12} {msg}")

clean: PASS
broken: 2 failures
   range        amount outside [0, 10000] on 135 rows
   cardinality  country has unseen values ['de', 'es', 'fr', 'gb', 'it']

Now count what it missed. The corruption multiplied 3,015 rows by a hundred, and the range check caught 135 of them — four and a half per cent. The other 2,880 are amounts between about 300 and 10,000, which are entirely plausible orders, and they sailed through every check in the suite. The cardinality check did its job loudly on the country column, and it only did so because the change there produced values that had never been seen before. Had the upstream release lower-cased nothing and only changed the units, the suite would have printed PASS on an extract in which one row in seven was wrong by a factor of a hundred.

Why silent corruption is the risk that matters#

A pipeline that crashes wakes someone up in an hour. A pipeline that quietly produces plausible nonsense trains a model, passes its evaluation on the same corrupted split, ships, and is discovered by a business metric drifting three months later. The cost is not the incident, it is the three months and everything built on top during them. Given the choice, prefer the loud failure every time: a schema that rejects an unexpected column is doing you a favour.

The way to catch the quiet version is to add a seventh check, on the shape of the distribution rather than on individual values. Summary statistics on the same corrupted column make the case for which statistic to use.

def psi(ref, cur, bins=10):
    edges = np.quantile(ref, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    r = np.clip(np.histogram(ref, edges)[0] / len(ref), 1e-6, None)
    c = np.clip(np.histogram(cur, edges)[0] / len(cur), 1e-6, None)
    return float(np.sum((c - r) * np.log(c / r)))

print("rows corrupted:", n_corrupt)
for stat, fn in [("median", lambda s: s.median()), ("mean", lambda s: s.mean()),
                 ("p99", lambda s: s.quantile(0.99))]:
    print(f"{stat:>7}  clean {fn(clean['amount']):>9.2f}   broken {fn(broken['amount']):>9.2f}")
print("PSI on amount:", round(psi(clean["amount"], broken["amount"]), 3))

rows corrupted: 3015
 median  clean     30.17   broken     35.00
   mean  clean     38.29   broken    622.66
    p99  clean    155.61   broken   8838.10
PSI on amount: 0.137

Which statistic you monitor decides whether you see this. The median is robust, which is exactly why it is useless here: it barely moves, because the corruption is a minority of rows. The mean and the upper percentiles are sensitive, which is normally a complaint about them and is the property you want in a monitor. PSI at 0.137 lands in the investigate-this band. So monitor a robust statistic to understand the typical case and a sensitive one to catch corruption, and never only the median.

What to alert on and what to merely record#

A suite like the one above produces a few dozen numbers per table per day, and a real pipeline has thirty tables. Alerting on all of it produces a channel nobody reads. The split that works is by what the receiver can do about it right now.

• Block the pipeline, and page: schema violations, freshness, and volume outside the baseline. These mean the data is wrong rather than surprising, and training or serving on it is worse than being late.
• Warn, and let the run continue: nullity and range failures on a small share of rows, unseen categories under a threshold. Somebody should look today, and stopping everything is a worse trade.
• Record only: every statistic in the suite, every day, for every column, whether it passed or not. This is the history that turns an investigation from an argument into a chart, and it costs a row in a table.
• Never alert on something with no owner and no action. An alert that arrives where nobody can act on it trains everyone to ignore the channel, including for the alerts that do have an action.

The recorded half is the part people skip and the part that pays off, because it is what tells you whether today's number is unusual. A nullity of 3 per cent means nothing on its own; 3 per cent against a 0.2 per cent history for ninety days is the whole finding. Store the statistics for everything and alert on almost none of it.

Once the suite is a page long, express it declaratively rather than as a function. Pandera is the smallest option and reads as a schema.

import pandera.pandas as pa

schema = pa.DataFrameSchema({
    "amount":   pa.Column(float, pa.Check.ge(0)),
    "country":  pa.Column(str, pa.Check.isin(sorted(COUNTRIES))),
    "label":    pa.Column(int, pa.Check.isin([0, 1])),
})
frame = pd.DataFrame({"amount": [12.0, 4.5], "country": ["GB", "DE"], "label": [0, 1]})
print(schema.validate(frame).shape)          # (2, 3): passes, returns the frame

try:
    schema.validate(pd.DataFrame({"amount": [-1.0], "country": ["ZZ"], "label": [0]}))
except pa.errors.SchemaError as e:
    print("raised:", type(e).__name__)        # raised: SchemaError

Look at slices#

An overall score can hide a group where the model is bad: one country, one device type, new customers. Break the evaluation down by the slices that matter to the business. Poor slices often turn out to be data problems, such as few examples or inconsistent labels in that group.