Model evaluation & metrics

Accuracy is the default answer and it is usually the wrong one. Which number you report should follow from what it costs to be wrong in each direction, and those costs are rarely equal: a missed fraud and a blocked customer are not the same mistake. This section starts from the confusion matrix, builds the common metrics out of it, and covers how to split data so the score you get is one you can believe.

What you are actually deciding              Report
a fixed number of cases get reviewed        precision at that capacity
a miss is far worse than a false alarm      recall at an acceptable alarm rate
both errors cost money you can estimate     expected cost at the threshold
you must compare two models, any threshold  ROC AUC, if classes are balanced
the same, but positives are under 5%        PR AUC
the probability itself is used downstream   log loss, and a calibration plot
there are many classes and all matter       macro F1, plus the confusion matrix

Start from the confusion matrix#

                  predicted positive   predicted negative
actually positive   true positive        false negative   (missed)
actually negative   false positive       true negative
                    (false alarm)

Build it once by hand and the metrics stop being formulae to memorise. Four counts, and each is a boolean pair.

import numpy as np

def confusion(y_true, y_pred):
    y_true, y_pred = np.asarray(y_true), np.asarray(y_pred)
    tp = int(((y_pred == 1) & (y_true == 1)).sum())
    fp = int(((y_pred == 1) & (y_true == 0)).sum())
    fn = int(((y_pred == 0) & (y_true == 1)).sum())
    tn = int(((y_pred == 0) & (y_true == 0)).sum())
    return tp, fp, fn, tn

# 1,000 transactions, 10 of them fraudulent. The model flags nothing.
y_true = np.zeros(1000, dtype=int)
y_true[:10] = 1
y_pred = np.zeros(1000, dtype=int)

tp, fp, fn, tn = confusion(y_true, y_pred)
print(tp, fp, fn, tn)                       # 0 0 10 990
print("accuracy", (tp + tn) / 1000)         # accuracy 0.99

The question that decides your metric is which mistake hurts more. Missing a fraudulent transaction and flagging a legitimate one are not equally bad, and no single number captures both.

Precision and recall#

Both are ratios of the same four counts, and the difference is only which denominator you use: precision divides by what you predicted, recall divides by what was true. Notice what each ignores. Precision never looks at the false negatives, so it cannot tell you what you missed. Recall never looks at the false positives, so it cannot tell you what the finding cost. Neither uses the true negatives at all, which is exactly why they survive on imbalanced data where accuracy does not.

def precision(tp, fp, fn, tn):
    return tp / (tp + fp) if tp + fp else 0.0        # of what I flagged

def recall(tp, fp, fn, tn):
    return tp / (tp + fn) if tp + fn else 0.0        # of what was there

def f1(tp, fp, fn, tn):
    p, r = precision(tp, fp, fn, tn), recall(tp, fp, fn, tn)
    return 2 * p * r / (p + r) if p + r else 0.0     # the harmonic mean

# Flag everything: perfect recall, and useless.
counts = (10, 990, 0, 0)
print(round(precision(*counts), 3), recall(*counts), round(f1(*counts), 3))
# 0.01 1.0 0.02

# A model that flags 20 and gets 8 of the 10.
counts = (8, 12, 2, 978)
print(round(precision(*counts), 2), round(recall(*counts), 2), round(f1(*counts), 3))
# 0.4 0.8 0.533

F1 is their harmonic mean rather than their ordinary average, and the choice is deliberate: the harmonic mean is dragged down by the smaller of the two, so a model cannot buy a good F1 by being excellent at one and hopeless at the other. Precision 1.0 with recall 0.02 averages to 0.51 the ordinary way and scores 0.039 as F1.

precision  recall   arithmetic mean   F1
1.00       0.02     0.51              0.039
0.60       0.60     0.60              0.600
0.90       0.30     0.60              0.450
0.40       0.80     0.60              0.533

Cancer screening      recall matters most, a miss is fatal
Spam filter           precision matters most, a lost email is worse
                      than a spam getting through
Fraud detection       depends on the cost of a review versus a loss

Thresholds#

A classifier outputs a probability, and the threshold turning that into a decision is yours to choose. Moving it trades precision against recall directly, and this is frequently a bigger lever than changing the model.

from sklearn.metrics import precision_recall_curve

precision, recall, thresholds = precision_recall_curve(y_true, y_scores)

# Pick the threshold meeting your business requirement,
# for example the highest recall while precision stays above 0.9

When you can put money on the two errors, skip the metric and minimise the cost directly. This is the answer that ends the argument about which metric to use, because it makes the weighting explicit instead of hiding it inside a formula.

import numpy as np

def expected_cost(y_true, scores, threshold, cost_fp, cost_fn):
    pred = scores >= threshold
    fp = int(((pred == 1) & (y_true == 0)).sum())
    fn = int(((pred == 0) & (y_true == 1)).sum())
    return fp * cost_fp + fn * cost_fn

rng = np.random.default_rng(0)
y = np.r_[np.zeros(2000), np.ones(100)].astype(int)
scores = np.r_[rng.normal(0.3, 0.15, 2000), rng.normal(0.6, 0.15, 100)]

# A review costs 5 pounds of someone's time; a missed fraud costs 200.
grid = np.linspace(0.1, 0.9, 33)
costs = [expected_cost(y, scores, t, cost_fp=5, cost_fn=200) for t in grid]
best = grid[int(np.argmin(costs))]
print(round(best, 2), min(costs))          # 0.35 4210

# Raise the cost of a miss to 500 and the best threshold falls: you accept more
# false alarms because each one is now cheaper relative to what it prevents.

AUC#

ROC AUC, the area under the ROC curve, summarises performance across all thresholds. It has a plain reading that is worth holding on to: it is the probability that a randomly chosen positive is scored above a randomly chosen negative. Half means the model has no ordering ability at all.

Precision-recall AUC, often reported as average precision, is the area under the precision-recall curve. Its baseline is not half; it is the prevalence of the positive class. On data that is one percent positive, a useless model scores 0.01, so any number has to be read against that.

import numpy as np

def roc_auc(y, s):
    """The rank formula, which is the definition rather than a curve integral."""
    order = np.argsort(s)
    ranks = np.empty(len(s))
    ranks[order] = np.arange(1, len(s) + 1)
    pos, neg = y.sum(), (1 - y).sum()
    return float((ranks[y == 1].sum() - pos * (pos + 1) / 2) / (pos * neg))

def average_precision(y, s):
    order = np.argsort(-s)
    yy = y[order]
    precision_at = np.cumsum(yy) / np.arange(1, len(yy) + 1)
    return float((precision_at * yy).sum() / yy.sum())

rng = np.random.default_rng(0)
y = np.r_[np.zeros(10_000), np.ones(100)].astype(int)          # 1% positive
s = np.r_[rng.normal(0, 1, 10_000), rng.normal(1.6, 1, 100)]   # a decent model

print("prevalence     ", round(float(y.mean()), 4))    # 0.0099
print("ROC AUC        ", round(roc_auc(y, s), 3))      # 0.857
print("PR AUC         ", round(average_precision(y, s), 3))   # 0.125

t = np.quantile(s[y == 1], 0.5)                        # threshold catching half
pred = s >= t
tp = int((pred & (y == 1)).sum())
fp = int((pred & (y == 0)).sum())
print("at half recall: ", tp, "caught,", fp, "false alarms, precision",
      round(tp / (tp + fp), 3))                        # 50 caught, 650, 0.071

• ROC AUC is right when the classes are roughly balanced, or when what you need is a single threshold-free comparison of ranking ability between models on the same data.
• ROC AUC lies when positives are rare. Its x-axis is the false positive rate, and with ten thousand negatives an extra six hundred false alarms moves that rate by 0.06 while destroying precision. It is optimistic in exactly the situation where you most need honesty.
• PR AUC is right when positives are rare and the cost of a false alarm is borne by a person who has to look at it.
• PR AUC lies when you compare across datasets or across time, because its baseline is the prevalence. A model whose PR AUC fell from 0.30 to 0.22 may be unchanged, with fraud simply having become rarer. Quote the prevalence beside it or the number means nothing.
• Both lie about calibration. A model can rank perfectly and output probabilities that are all far too high, and both AUCs will be one. If the probability is used for anything beyond ordering, score it with log loss and look at a calibration plot.

Cross-validation#

A single train and test split gives one estimate with real variance. K-fold cross-validation rotates the split and averages, giving a more stable number and a sense of how much it varies. Report the spread as well as the mean: five folds scoring 0.81 with a standard deviation of 0.01 and five folds averaging 0.81 with a deviation of 0.09 are different situations, and only the second one explains why your colleague cannot reproduce your number.

Situation                              Split to use
ordinary tabular data                  KFold, shuffled
imbalanced classes                     StratifiedKFold, so every fold has positives
several rows per patient, user, shop   GroupKFold on that id
anything with a time order             TimeSeriesSplit, never shuffled
tuning hyperparameters AND reporting   nested: an inner loop tunes, an outer scores

The grouped case is the one that catches experienced people. If the same customer appears in twenty rows and those rows are split across folds, the model has seen that customer's behaviour during training and is asked about them in validation. The score is about memorising customers, and the model meets only new ones in production. Group on whatever the repeated entity is, and if you are not sure whether there is one, count distinct ids against rows before you split.

Nested cross-validation is the other one people skip. If you use cross-validation to choose hyperparameters and then report the best cross-validated score, that number is optimistic: you selected the maximum of many noisy estimates, and the maximum of noise is above the mean of noise. With fifty configurations tried, the winner is typically a point or two better than it deserves. Either use an inner loop for tuning and an outer loop for reporting, or keep a test set that nothing has been selected on.

Leakage#

Being warned about leakage does not help you notice it, so here it is happening. The target below is a coin flip. Nothing in the data predicts it, and the honest score is 0.5. One feature is added: the average target for each user, computed over the whole dataset before splitting — an ordinary target encoding, written the way it is usually first written.

import numpy as np

rng = np.random.default_rng(1)
n = 300
y = rng.integers(0, 2, n)                 # a coin flip: unpredictable by construction
user = rng.integers(0, 100, n)            # 100 users, so about 3 rows each
folds = np.arange(n) % 5

# The leak: each user's mean target, computed over everything, including this row.
means = np.array([y[user == u].mean() if (user == u).any() else 0.5 for u in range(100)])
leaky = means[user]

def accuracy(feature, mask):
    return ((feature[mask] > 0.5) == y[mask]).mean()

print("cross-validated:", round(np.mean([accuracy(leaky, folds == f) for f in range(5)]), 3))
# cross-validated: 0.7

# The same feature computed out of fold: each row's encoding uses only other folds.
honest = np.empty(n)
for f in range(5):
    tr = folds != f
    m = np.array([
        y[tr & (user == u)].mean() if (tr & (user == u)).any() else y[tr].mean()
        for u in range(100)
    ])
    honest[folds == f] = m[user[folds == f]]

print("out of fold:    ", round(np.mean([accuracy(honest, folds == f) for f in range(5)]), 3))
# out of fold:     0.5

That is the general shape of it: the leak happens before the split, so the split cannot see it. Every transformation that learns anything from the data — a scaler's mean, an imputer's median, a target encoding, a feature selector's ranking, an oversampler — must be fitted inside the fold. A pipeline is how you enforce that mechanically rather than by remembering.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

# Wrong: the scaler has seen the test data
X_scaled = StandardScaler().fit_transform(X)
train_test_split(X_scaled, y)

# Right: the scaler is fitted inside each fold
pipe = make_pipeline(StandardScaler(), LogisticRegression())
cross_val_score(pipe, X, y, cv=5)

• The tell is a score that is too good. A jump of fifteen points from one new feature is a leak until proven otherwise, and proving otherwise takes an hour that is always worth spending.
• The second tell is a feature at the top of the importance list that nobody can explain. Ask when its value becomes known. If the answer is after the thing you are predicting, it is a leak.
• The third is a gap between a cross-validated score and a genuinely held-out one. Keep a final test set that no decision has touched, and look at it once.

Classification metrics get most of the attention, but plenty of what gets predicted is a number: a demand, a duration, a price. This section covers the metrics for that and, more usefully, what each one chooses to punish — RMSE cares about large errors, MAE treats them all alike, MAPE falls apart near zero. The second half is what decides whether your work gets used: translating any of these into an effect somebody outside the team cares about, because the model with the better score can still be the worse one if its errors land on the days that matter.

Regression metrics#

• Mean absolute error: average size of the error. Easy to interpret, treats all errors equally.
• Root mean squared error: penalises large errors heavily. Use it when a big miss is much worse than several small ones.
• Mean absolute percentage error: relative error. Intuitive, and it breaks when actual values are near zero.

Errors:  1, 1, 1, 1, 10

MAE  = 2.8     the typical error
RMSE = 4.6     pulled up by the single large miss

That gap is a diagnostic worth using deliberately. RMSE is always at least MAE, and the ratio between them tells you the shape of your error distribution: close to one means the errors are all much the same size, and a ratio of two or more means a handful of large misses are carrying the number. Which you should care about depends entirely on whether one big miss costs more than five small ones, and that is a question about the business, not about the model.

import numpy as np

def mae(y, p):
    return float(np.mean(np.abs(y - p)))

def rmse(y, p):
    return float(np.sqrt(np.mean((y - p) ** 2)))

def mape(y, p):
    return float(np.mean(np.abs((y - p) / y)))          # undefined at y = 0

def wape(y, p):
    """Weighted absolute percentage error: total error over total actual.
    Survives zeros, and is what most demand planners actually want."""
    return float(np.abs(y - p).sum() / np.abs(y).sum())

y = np.array([100.0, 100, 100, 100, 100])
spread = np.array([98.0, 102, 99, 101, 100])            # five small misses
lumpy = np.array([100.0, 100, 100, 100, 90])            # one large miss

for name, p in [("spread", spread), ("lumpy", lumpy)]:
    print(name, "MAE", round(mae(y, p), 2), "RMSE", round(rmse(y, p), 2),
          "ratio", round(rmse(y, p) / mae(y, p), 2))

# spread MAE 1.2 RMSE 1.41 ratio 1.18   errors all alike
# lumpy  MAE 2.0 RMSE 4.47 ratio 2.24   one miss is doing the work

MAPE deserves its own warning because it is the one most often asked for by name. It divides by the actual, so it is undefined at zero and enormous near it, which is fatal for anything intermittent: a product that sells nothing on Tuesday cannot have a percentage error on Tuesday. Worse, it is asymmetric. Over-predicting by fifty when the actual is a hundred costs 0.5; under-predicting to zero also costs 1.0 at most, whatever the over-prediction, because the error is capped by the actual on one side and unbounded on the other. A model tuned on MAPE learns to forecast low.

import numpy as np

actual = 100.0
for pred in (50.0, 150.0, 0.0, 200.0):
    print(pred, round(abs(actual - pred) / actual, 2))

# 50.0  0.5     under by half
# 150.0 0.5     over by half, same penalty, so far so fair
# 0.0   1.0     predict nothing: the worst possible under-prediction costs 1.0
# 200.0 1.0     predict double: the same 1.0, and there is no ceiling above it

print(round(np.mean([abs(0 - 5) / 5, abs(10 - 5) / 5]), 2))  # 1.0
print(round(np.mean([abs(0.1 - 0.0001) / 0.0001, 0]), 1))    # 499.5 from one tiny actual

When the two directions genuinely cost different amounts, stop trying to find a symmetric metric that captures it and use a loss that does not pretend. The pinball, or quantile, loss weights over- and under-prediction differently by construction, and minimising it produces a forecast of that quantile rather than the mean.

import numpy as np

def pinball(y, p, q):
    """q is the quantile. q=0.9 punishes under-prediction nine times as hard."""
    error = y - p
    return float(np.mean(np.maximum(q * error, (q - 1) * error)))

y = np.array([100.0] * 5)
print(round(pinball(y, np.full(5, 90.0), 0.9), 2))    # 9.0   under by 10
print(round(pinball(y, np.full(5, 110.0), 0.9), 2))   # 1.0   over by 10, far cheaper
print(round(pinball(y, np.full(5, 90.0), 0.5), 2))    # 5.0   at q=0.5 this is MAE/2

Connect to what matters#

A metric is only useful if it moves with something the business cares about. A forecasting model with lower RMSE may still perform worse in practice if its errors fall on the days that matter most, such as the busiest period.

The way to settle that is to compute the thing itself rather than a proxy for it. Attach a cost to each direction of error, sum it over the test period, and compare models on pounds. It takes an afternoon, it usually reorders the models, and it turns an argument about metrics into an arithmetic problem.

import numpy as np

def stock_cost(actual, forecast, holding=1.0, stockout=6.0):
    """Over-forecast: units sit in a warehouse. Under-forecast: a lost sale."""
    over = np.maximum(forecast - actual, 0).sum() * holding
    under = np.maximum(actual - forecast, 0).sum() * stockout
    return float(over + under)

rng = np.random.default_rng(0)
actual = rng.poisson(40, 365).astype(float)

cautious = np.full(365, 40.0)                    # the mean: the RMSE-optimal answer
padded = np.full(365, 48.0)                      # deliberately over-forecast

for name, f in [("mean forecast", cautious), ("padded forecast", padded)]:
    print(f"{name:16}",
          "RMSE", round(float(np.sqrt(np.mean((actual - f) ** 2))), 2),
          "| cost", round(stock_cost(actual, f)))

# mean forecast    RMSE 6.51 | cost 7310
# padded forecast  RMSE 9.77 | cost 3762

Accuracy is one number, and it hides nearly everything worth knowing. This section starts from the counts underneath it — what was predicted against what was true — and builds up the metrics that tell the kinds of mistake apart. It covers the averaging choices that can change a multi-class result completely, scoring the probabilities a model outputs rather than only the label it picked, and computing metrics per slice, which is usually where a model that looks fine turns out not to be.

Start from the counts#

from sklearn.metrics import confusion_matrix, classification_report

y_true = ["billing", "bug", "billing", "account", "bug", "billing", "account", "bug"]
y_pred = ["billing", "bug", "bug",     "account", "bug", "billing", "billing", "bug"]

print(confusion_matrix(y_true, y_pred, labels=["account", "billing", "bug"]))
print(classification_report(y_true, y_pred))

Every metric is a summary of this table. Read the table first; the summary hides which classes get confused with which. Off the diagonal is where the information is: a model confusing billing with account is a labelling problem, and a model confusing everything with the most common class is a class-imbalance problem, and the single number cannot tell you which you have.

Averaging across classes#

• Macro: compute the metric per class and average them equally. A rare class counts as much as a common one. Use it when every class matters.
• Weighted: average per-class metrics weighted by how many examples each has. Dominated by common classes.
• Micro: pool all decisions, then compute. For single-label classification, micro F1 equals accuracy.

Those three sentences are easy to read and hard to feel. Build the table by hand once, on a support ticket classifier where the rare class is never predicted at all, and the difference stops being abstract.

import numpy as np

labels = ["account", "billing", "bug"]
y_true = np.array([0] * 5 + [1] * 35 + [2] * 60)       # account is 5% of tickets
y_pred = np.array([1] * 5 + [1] * 30 + [2] * 5 + [2] * 60)   # account is never predicted

def confusion_matrix(y_true, y_pred, n):
    m = np.zeros((n, n), dtype=int)
    for t, p in zip(y_true, y_pred):
        m[t, p] += 1                                   # rows true, columns predicted
    return m

def per_class(M):
    tp = np.diag(M).astype(float)
    fp = M.sum(axis=0) - tp                            # column total minus the diagonal
    fn = M.sum(axis=1) - tp                            # row total minus the diagonal
    z = np.zeros_like(tp)
    prec = np.divide(tp, tp + fp, out=z.copy(), where=(tp + fp) > 0)
    rec = np.divide(tp, tp + fn, out=z.copy(), where=(tp + fn) > 0)
    f1 = np.divide(2 * prec * rec, prec + rec, out=z.copy(), where=(prec + rec) > 0)
    return prec, rec, f1, M.sum(axis=1)

M = confusion_matrix(y_true, y_pred, 3)
print(M)
prec, rec, f1, support = per_class(M)
for i, name in enumerate(labels):
    print(f"{name:9} precision {prec[i]:.3f} recall {rec[i]:.3f} f1 {f1[i]:.3f} n {support[i]}")

print("macro   ", round(float(f1.mean()), 3))
print("weighted", round(float((f1 * support).sum() / support.sum()), 3))
print("micro   ", round(float(np.diag(M).sum() / M.sum()), 3), "(= accuracy)")

# [[ 0  5  0]
#  [ 0 30  5]
#  [ 0  0 60]]
# account   precision 0.000 recall 0.000 f1 0.000 n 5
# billing   precision 0.857 recall 0.857 f1 0.857 n 35
# bug       precision 0.923 recall 1.000 f1 0.960 n 60
# macro    0.606
# weighted 0.876
# micro    0.9 (= accuracy)

from sklearn.metrics import f1_score

f1_score(y_true, y_pred, average="macro")
f1_score(y_true, y_pred, average="weighted")

Score the probabilities#

Accuracy only checks whether the top guess was right. If the probabilities are used, for pricing risk or deciding when to ask a person, score them directly. Log loss punishes confident wrong answers heavily; the Brier score is the mean squared error of the probabilities and is easier to interpret.

from sklearn.metrics import log_loss, brier_score_loss

log_loss(y_binary, p_positive)          # lower is better; confident mistakes cost a lot
brier_score_loss(y_binary, p_positive)  # 0 is perfect; always predicting 0.5 gives 0.25

Neither number tells you the shape of the problem, and the shape is what you fix. For that, bin the predictions and compare what the model said against what actually happened in each bin. A well-calibrated model's two columns match.

import numpy as np

def calibration(y, p, bins=5):
    edges = np.linspace(0, 1, bins + 1)
    idx = np.clip(np.digitize(p, edges[1:-1]), 0, bins - 1)
    for b in range(bins):
        m = idx == b
        if m.sum():
            print(f"{edges[b]:.1f}-{edges[b+1]:.1f}  n={int(m.sum()):5d}"
                  f"  said {p[m].mean():.3f}  happened {y[m].mean():.3f}")

rng = np.random.default_rng(0)
n = 4000
true_p = rng.uniform(0.05, 0.95, n)
y = (rng.random(n) < true_p).astype(int)
overconfident = true_p ** 2 / (true_p ** 2 + (1 - true_p) ** 2)   # pushed to the extremes

calibration(y, overconfident)
# 0.0-0.2  n= 1243  said 0.067  happened 0.195
# 0.2-0.4  n=  574  said 0.296  happened 0.415
# 0.4-0.6  n=  428  said 0.501  happened 0.505
# 0.6-0.8  n=  512  said 0.703  happened 0.635
# 0.8-1.0  n= 1243  said 0.930  happened 0.808

Per slice#

df = X_test.assign(y=y_test, pred=model.predict(X_test))
for group, part in df.groupby("country"):
    print(group, len(part), f1_score(part.y, part.pred, average="macro"))

Which slices, decided before you look: the ones where the population differs, the ones where a failure is expensive, and the ones that are small enough to be invisible in the average. In practice that usually means country or language, new users against established ones, mobile against desktop, and the most recent week against the rest. Anything with fewer than a few hundred examples in a slice has an error bar wider than most of the differences you will be tempted to act on, so compute the count alongside the score and say so.

Slice averages can also reverse the overall result. A model can be worse than another on every single segment and better overall, if the segments are different sizes and the two models see different mixes. When an aggregate and a per-slice story disagree, the per-slice one is usually the one that describes what users experience.

Computing a metric is not the same as knowing which one to optimise, and that choice comes from the problem rather than from a list. This section reasons through four cases where the costs are genuinely different — a spam filter, cancer screening, loan defaults, search results — starting each time from what happens after a prediction, what each kind of error costs, and who pays it. The method transfers to your own problem; the four answers do not.

The method#

Ask what happens after each prediction, what each kind of error costs, and who pays. The metric follows from the answers; it is not chosen from a list.

1  What happens next?      Who or what acts on this prediction, automatically
                           or with a person in the loop?
2  What is the capacity?   How many positives can be acted on per day? A fixed
                           budget turns the question into precision at that k.
3  What does each error    In the units the business already uses: pounds,
   cost, and to whom?      minutes of someone's time, a lost customer.
4  Is the probability      If it is multiplied by anything downstream, you need
   used, or only the       calibration and log loss, not only ranking.
   ordering?

Spam filter        false positive: a real email lost        costly
                   false negative: a spam in the inbox       mildly annoying
                   -> high precision on 'spam'; accept lower recall

Cancer screening   false negative: a missed cancer          very costly
                   false positive: a follow-up test         cheaper
                   -> high recall at an acceptable follow-up rate

Loan default       both errors cost money, in amounts you can estimate
                   -> expected profit at a threshold, from calibrated probabilities

Search results     users look at the top few only
                   -> ranking metrics at small k: precision@5, NDCG@10

The spam filter, in numbers#

The asymmetry here is extreme and worth quantifying, because it explains an operating point that looks absurd from the outside. A spam in the inbox costs someone two seconds. A legitimate invoice silently filed in the junk folder costs a business relationship. Tolerances in real filters are of the order of one legitimate message misfiled in ten thousand, which on a mailbox receiving a hundred good messages a day is one mistake every three years.

That sets the metric: recall on spam, measured at a false positive rate of 0.0001. Not F1, which would happily trade a lost invoice for several caught spams. And it sets what you must measure: at that rate, a test set of a thousand legitimate emails contains an expected 0.1 false positives, so it cannot distinguish a good filter from a terrible one. The evaluation set has to be large enough for the rate you care about, which for one in ten thousand means hundreds of thousands of legitimate messages.

Cancer screening, in numbers#

Run the arithmetic on a screened population of a hundred thousand where the condition is present in five hundred of them. Take a model at ninety percent recall and a ten percent false positive rate, which would be a respectable classifier on most problems.

                        has it (500)     does not (99,500)
flagged                    450                9,950
not flagged                 50               89,550

recall     450 / 500       = 0.90     the number the programme is designed around
precision  450 / 10,400    = 0.043    96 percent of the people recalled are well
accuracy   90,000 / 100,000 = 0.90    identical to the recall, and meaningless here

So the metric is recall at a fixed false positive rate, with the rate set by follow-up capacity rather than by the model. Moving recall from 0.90 to 0.95 sounds small and means fifty more people helped and, at this operating point, several thousand more recalled. Whether that is the right trade is a question for the people who run the programme, and the model's job is to present it as a curve rather than a single number.

Loan defaults, in numbers#

This is the case where there is a right answer and it is arithmetic. Both errors cost money, the amounts are known, so the metric is expected profit and the threshold falls out of the two costs rather than being tuned.

def expected_profit(p_default, gain=1200.0, loss=8000.0):
    """gain: interest earned if the loan is repaid.
    loss: the amount written off if it is not."""
    return (1 - p_default) * gain - p_default * loss

for p in (0.05, 0.10, 0.13, 0.15, 0.30):
    print(p, round(expected_profit(p), 1))

print("break even at p =", round(1200 / (1200 + 8000), 4))

# 0.05   740.0
# 0.1    280.0
# 0.13     4.0
# 0.15  -180.0
# 0.3  -1560.0
# break even at p = 0.1304

The catch is in the word probability. This only works if a predicted 0.13 means thirteen percent of such loans default, so the model must be calibrated and not merely well-ranked. That is why logistic regression and calibrated gradient boosting persist in credit scoring: the number is used arithmetically, so ROC AUC is the wrong thing to report and log loss with a calibration table is the right one.

Search results, in numbers#

Users look at the first few results and abandon rather than scroll, so anything computed over a hundred results is measuring something nobody sees. Report NDCG at 10 as the headline and precision at 3 alongside it, because the first three are the ones that decide whether the user tries again or leaves.

Two guardrails belong next to it. Latency, because a result set that is two percent better and two hundred milliseconds slower is worse in every measurable way. And the share of queries returning nothing useful at all, which an average over all queries hides completely: a system with a good mean NDCG and eight percent of queries returning rubbish feels broken, because the queries that fail are the ones people remember.

What transfers#

• The metric is downstream of the decision. Name the action and the capacity first, and the metric is usually forced.
• One number is never enough. Report the pair that can be traded against each other, plus the guardrail that stops someone gaming it.
• If the probability is used arithmetically, calibration is not optional and ranking metrics are silent about it.
• Make sure the test set is large enough for the operating point. A rate of one in ten thousand cannot be measured on a thousand examples, and people try.