Choosing the metric that matches the cost of being wrong
By the end: You can pick and defend an evaluation metric, which is a standard interview question.
- The confusion matrix
- Precision, recall and F1
- ROC AUC and precision-recall AUC
- Cross-validation and leakage
New to model evaluation & metrics? Once a model makes predictions you need a number saying how good they are. Evaluation is choosing that number and measuring it honestly. Accuracy — the share of predictions that were right — is the obvious choice and often the wrong one: a test for a disease that one person in a thousand has is 99.9% accurate if it simply says no to everybody. Why it matters
Accuracy is the default answer and it is usually the wrong one. Which number you report should follow from what it costs to be wrong in each direction, and those costs are rarely equal: a missed fraud and a blocked customer are not the same mistake. This section starts from the confusion matrix, builds the common metrics out of it, and covers how to split data so the score you get is one you can believe.
What you are actually deciding Report
a fixed number of cases get reviewed precision at that capacity
a miss is far worse than a false alarm recall at an acceptable alarm rate
both errors cost money you can estimate expected cost at the threshold
you must compare two models, any threshold ROC AUC, if classes are balanced
the same, but positives are under 5% PR AUC
the probability itself is used downstream log loss, and a calibration plot
there are many classes and all matter macro F1, plus the confusion matrixStart from the confusion matrix#
predicted positive predicted negative
actually positive true positive false negative (missed)
actually negative false positive true negative
(false alarm)Build it once by hand and the metrics stop being formulae to memorise. Four counts, and each is a boolean pair.
import numpy as np
def confusion(y_true, y_pred):
y_true, y_pred = np.asarray(y_true), np.asarray(y_pred)
tp = int(((y_pred == 1) & (y_true == 1)).sum())
fp = int(((y_pred == 1) & (y_true == 0)).sum())
fn = int(((y_pred == 0) & (y_true == 1)).sum())
tn = int(((y_pred == 0) & (y_true == 0)).sum())
return tp, fp, fn, tn
# 1,000 transactions, 10 of them fraudulent. The model flags nothing.
y_true = np.zeros(1000, dtype=int)
y_true[:10] = 1
y_pred = np.zeros(1000, dtype=int)
tp, fp, fn, tn = confusion(y_true, y_pred)
print(tp, fp, fn, tn) # 0 0 10 990
print("accuracy", (tp + tn) / 1000) # accuracy 0.99The question that decides your metric is which mistake hurts more. Missing a fraudulent transaction and flagging a legitimate one are not equally bad, and no single number captures both.
Precision and recall#
Both are ratios of the same four counts, and the difference is only which denominator you use: precision divides by what you predicted, recall divides by what was true. Notice what each ignores. Precision never looks at the false negatives, so it cannot tell you what you missed. Recall never looks at the false positives, so it cannot tell you what the finding cost. Neither uses the true negatives at all, which is exactly why they survive on imbalanced data where accuracy does not.
def precision(tp, fp, fn, tn):
return tp / (tp + fp) if tp + fp else 0.0 # of what I flagged
def recall(tp, fp, fn, tn):
return tp / (tp + fn) if tp + fn else 0.0 # of what was there
def f1(tp, fp, fn, tn):
p, r = precision(tp, fp, fn, tn), recall(tp, fp, fn, tn)
return 2 * p * r / (p + r) if p + r else 0.0 # the harmonic mean
# Flag everything: perfect recall, and useless.
counts = (10, 990, 0, 0)
print(round(precision(*counts), 3), recall(*counts), round(f1(*counts), 3))
# 0.01 1.0 0.02
# A model that flags 20 and gets 8 of the 10.
counts = (8, 12, 2, 978)
print(round(precision(*counts), 2), round(recall(*counts), 2), round(f1(*counts), 3))
# 0.4 0.8 0.533F1 is their harmonic mean rather than their ordinary average, and the choice is deliberate: the harmonic mean is dragged down by the smaller of the two, so a model cannot buy a good F1 by being excellent at one and hopeless at the other. Precision 1.0 with recall 0.02 averages to 0.51 the ordinary way and scores 0.039 as F1.
precision recall arithmetic mean F1
1.00 0.02 0.51 0.039
0.60 0.60 0.60 0.600
0.90 0.30 0.60 0.450
0.40 0.80 0.60 0.533Cancer screening recall matters most, a miss is fatal
Spam filter precision matters most, a lost email is worse
than a spam getting through
Fraud detection depends on the cost of a review versus a lossWhere F1 lies
It weights precision and recall equally, and almost no real problem does. If a miss costs ten times a false alarm, a model tuned to maximise F1 is tuned to the wrong thing and will look like the best of the set on your table. F1 also never touches the true negatives, so it is silent about a model that flags a third of your traffic. Use it to compare models on the same problem, not to choose an operating point.
Thresholds#
A classifier outputs a probability, and the threshold turning that into a decision is yours to choose. Moving it trades precision against recall directly, and this is frequently a bigger lever than changing the model.
from sklearn.metrics import precision_recall_curve
precision, recall, thresholds = precision_recall_curve(y_true, y_scores)
# Pick the threshold meeting your business requirement,
# for example the highest recall while precision stays above 0.9When you can put money on the two errors, skip the metric and minimise the cost directly. This is the answer that ends the argument about which metric to use, because it makes the weighting explicit instead of hiding it inside a formula.
import numpy as np
def expected_cost(y_true, scores, threshold, cost_fp, cost_fn):
pred = scores >= threshold
fp = int(((pred == 1) & (y_true == 0)).sum())
fn = int(((pred == 0) & (y_true == 1)).sum())
return fp * cost_fp + fn * cost_fn
rng = np.random.default_rng(0)
y = np.r_[np.zeros(2000), np.ones(100)].astype(int)
scores = np.r_[rng.normal(0.3, 0.15, 2000), rng.normal(0.6, 0.15, 100)]
# A review costs 5 pounds of someone's time; a missed fraud costs 200.
grid = np.linspace(0.1, 0.9, 33)
costs = [expected_cost(y, scores, t, cost_fp=5, cost_fn=200) for t in grid]
best = grid[int(np.argmin(costs))]
print(round(best, 2), min(costs)) # 0.35 4210
# Raise the cost of a miss to 500 and the best threshold falls: you accept more
# false alarms because each one is now cheaper relative to what it prevents.AUC#
ROC AUC, the area under the ROC curve, summarises performance across all thresholds. It has a plain reading that is worth holding on to: it is the probability that a randomly chosen positive is scored above a randomly chosen negative. Half means the model has no ordering ability at all.
Precision-recall AUC, often reported as average precision, is the area under the precision-recall curve. Its baseline is not half; it is the prevalence of the positive class. On data that is one percent positive, a useless model scores 0.01, so any number has to be read against that.
import numpy as np
def roc_auc(y, s):
"""The rank formula, which is the definition rather than a curve integral."""
order = np.argsort(s)
ranks = np.empty(len(s))
ranks[order] = np.arange(1, len(s) + 1)
pos, neg = y.sum(), (1 - y).sum()
return float((ranks[y == 1].sum() - pos * (pos + 1) / 2) / (pos * neg))
def average_precision(y, s):
order = np.argsort(-s)
yy = y[order]
precision_at = np.cumsum(yy) / np.arange(1, len(yy) + 1)
return float((precision_at * yy).sum() / yy.sum())
rng = np.random.default_rng(0)
y = np.r_[np.zeros(10_000), np.ones(100)].astype(int) # 1% positive
s = np.r_[rng.normal(0, 1, 10_000), rng.normal(1.6, 1, 100)] # a decent model
print("prevalence ", round(float(y.mean()), 4)) # 0.0099
print("ROC AUC ", round(roc_auc(y, s), 3)) # 0.857
print("PR AUC ", round(average_precision(y, s), 3)) # 0.125
t = np.quantile(s[y == 1], 0.5) # threshold catching half
pred = s >= t
tp = int((pred & (y == 1)).sum())
fp = int((pred & (y == 0)).sum())
print("at half recall: ", tp, "caught,", fp, "false alarms, precision",
round(tp / (tp + fp), 3)) # 50 caught, 650, 0.071- ROC AUC is right when the classes are roughly balanced, or when what you need is a single threshold-free comparison of ranking ability between models on the same data.
- ROC AUC lies when positives are rare. Its x-axis is the false positive rate, and with ten thousand negatives an extra six hundred false alarms moves that rate by 0.06 while destroying precision. It is optimistic in exactly the situation where you most need honesty.
- PR AUC is right when positives are rare and the cost of a false alarm is borne by a person who has to look at it.
- PR AUC lies when you compare across datasets or across time, because its baseline is the prevalence. A model whose PR AUC fell from 0.30 to 0.22 may be unchanged, with fraud simply having become rarer. Quote the prevalence beside it or the number means nothing.
- Both lie about calibration. A model can rank perfectly and output probabilities that are all far too high, and both AUCs will be one. If the probability is used for anything beyond ordering, score it with log loss and look at a calibration plot.
Cross-validation#
A single train and test split gives one estimate with real variance. K-fold cross-validation rotates the split and averages, giving a more stable number and a sense of how much it varies. Report the spread as well as the mean: five folds scoring 0.81 with a standard deviation of 0.01 and five folds averaging 0.81 with a deviation of 0.09 are different situations, and only the second one explains why your colleague cannot reproduce your number.
Situation Split to use
ordinary tabular data KFold, shuffled
imbalanced classes StratifiedKFold, so every fold has positives
several rows per patient, user, shop GroupKFold on that id
anything with a time order TimeSeriesSplit, never shuffled
tuning hyperparameters AND reporting nested: an inner loop tunes, an outer scoresThe grouped case is the one that catches experienced people. If the same customer appears in twenty rows and those rows are split across folds, the model has seen that customer's behaviour during training and is asked about them in validation. The score is about memorising customers, and the model meets only new ones in production. Group on whatever the repeated entity is, and if you are not sure whether there is one, count distinct ids against rows before you split.
Nested cross-validation is the other one people skip. If you use cross-validation to choose hyperparameters and then report the best cross-validated score, that number is optimistic: you selected the maximum of many noisy estimates, and the maximum of noise is above the mean of noise. With fifty configurations tried, the winner is typically a point or two better than it deserves. Either use an inner loop for tuning and an outer loop for reporting, or keep a test set that nothing has been selected on.
Leakage#
Leakage is the mistake that matters most
If information from the test set influences training, your score is fiction. It happens through scaling fitted on all the data before splitting, features computed using future information, duplicate rows across splits, and shuffled splits on time series. Every one of these produces an excellent score and a model that fails in production.
Being warned about leakage does not help you notice it, so here it is happening. The target below is a coin flip. Nothing in the data predicts it, and the honest score is 0.5. One feature is added: the average target for each user, computed over the whole dataset before splitting — an ordinary target encoding, written the way it is usually first written.
import numpy as np
rng = np.random.default_rng(1)
n = 300
y = rng.integers(0, 2, n) # a coin flip: unpredictable by construction
user = rng.integers(0, 100, n) # 100 users, so about 3 rows each
folds = np.arange(n) % 5
# The leak: each user's mean target, computed over everything, including this row.
means = np.array([y[user == u].mean() if (user == u).any() else 0.5 for u in range(100)])
leaky = means[user]
def accuracy(feature, mask):
return ((feature[mask] > 0.5) == y[mask]).mean()
print("cross-validated:", round(np.mean([accuracy(leaky, folds == f) for f in range(5)]), 3))
# cross-validated: 0.7
# The same feature computed out of fold: each row's encoding uses only other folds.
honest = np.empty(n)
for f in range(5):
tr = folds != f
m = np.array([
y[tr & (user == u)].mean() if (tr & (user == u)).any() else y[tr].mean()
for u in range(100)
])
honest[folds == f] = m[user[folds == f]]
print("out of fold: ", round(np.mean([accuracy(honest, folds == f) for f in range(5)]), 3))
# out of fold: 0.5That is the general shape of it: the leak happens before the split, so the split cannot see it. Every transformation that learns anything from the data — a scaler's mean, an imputer's median, a target encoding, a feature selector's ranking, an oversampler — must be fitted inside the fold. A pipeline is how you enforce that mechanically rather than by remembering.
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
# Wrong: the scaler has seen the test data
X_scaled = StandardScaler().fit_transform(X)
train_test_split(X_scaled, y)
# Right: the scaler is fitted inside each fold
pipe = make_pipeline(StandardScaler(), LogisticRegression())
cross_val_score(pipe, X, y, cv=5)- The tell is a score that is too good. A jump of fifteen points from one new feature is a leak until proven otherwise, and proving otherwise takes an hour that is always worth spending.
- The second tell is a feature at the top of the importance list that nobody can explain. Ask when its value becomes known. If the answer is after the thing you are predicting, it is a leak.
- The third is a gap between a cross-validated score and a genuinely held-out one. Keep a final test set that no decision has touched, and look at it once.
Where people go wrong
Reporting accuracy on imbalanced data. A model predicting the majority class every time scores very well and is worthless, and the number gives no hint of that.
If this comes up in an interview
When would you optimise for recall over precision?
When a miss costs more than a false alarm: disease screening, fraud, safety. When a false alarm is expensive, such as a spam filter deleting real mail, precision matters more. The metric follows from the asymmetry of the costs.
Why can ROC AUC mislead on imbalanced data?
It includes the true negative rate, and with overwhelming negatives that stays high regardless, flattering the score. Precision-recall AUC ignores true negatives and is the honest summary when positives are rare.
What is data leakage and how does it happen?
Information from the test set influencing training. Common routes are fitting a scaler or imputer before splitting, features built from future information, duplicate rows across splits, and shuffling a time series. All of them inflate the score and produce a model that fails in production.
How do you choose a classification threshold?
From the business requirement rather than the default of 0.5. Plot the precision-recall curve and pick the point meeting the constraint, such as maximum recall while precision stays above a required level.
Build this: Train a classifier on imbalanced data and report accuracy, precision, recall, F1, ROC AUC and PR AUC. Then move the threshold and watch which numbers move together and which trade against each other.
Watch it explained
This is one section of Model evaluation & metrics, 16 hours across 4 sections. Read it in full to get the whole topic in order.