Metrics in code, with the traps
By the end: You can compute the standard metrics in scikit-learn, choose the right averaging for several classes, and score probabilities rather than only labels.
- A confusion matrix from predictions
- Macro, micro and weighted averages
- Scoring probabilities: log loss and Brier score
- Metrics per slice
New to model evaluation & metrics? Once a model makes predictions you need a number saying how good they are. Evaluation is choosing that number and measuring it honestly. Accuracy — the share of predictions that were right — is the obvious choice and often the wrong one: a test for a disease that one person in a thousand has is 99.9% accurate if it simply says no to everybody. Why it matters
Accuracy is one number, and it hides nearly everything worth knowing. This section starts from the counts underneath it — what was predicted against what was true — and builds up the metrics that tell the kinds of mistake apart. It covers the averaging choices that can change a multi-class result completely, scoring the probabilities a model outputs rather than only the label it picked, and computing metrics per slice, which is usually where a model that looks fine turns out not to be.
Start from the counts#
from sklearn.metrics import confusion_matrix, classification_report
y_true = ["billing", "bug", "billing", "account", "bug", "billing", "account", "bug"]
y_pred = ["billing", "bug", "bug", "account", "bug", "billing", "billing", "bug"]
print(confusion_matrix(y_true, y_pred, labels=["account", "billing", "bug"]))
print(classification_report(y_true, y_pred))Every metric is a summary of this table. Read the table first; the summary hides which classes get confused with which. Off the diagonal is where the information is: a model confusing billing with account is a labelling problem, and a model confusing everything with the most common class is a class-imbalance problem, and the single number cannot tell you which you have.
Averaging across classes#
- Macro: compute the metric per class and average them equally. A rare class counts as much as a common one. Use it when every class matters.
- Weighted: average per-class metrics weighted by how many examples each has. Dominated by common classes.
- Micro: pool all decisions, then compute. For single-label classification, micro F1 equals accuracy.
Those three sentences are easy to read and hard to feel. Build the table by hand once, on a support ticket classifier where the rare class is never predicted at all, and the difference stops being abstract.
import numpy as np
labels = ["account", "billing", "bug"]
y_true = np.array([0] * 5 + [1] * 35 + [2] * 60) # account is 5% of tickets
y_pred = np.array([1] * 5 + [1] * 30 + [2] * 5 + [2] * 60) # account is never predicted
def confusion_matrix(y_true, y_pred, n):
m = np.zeros((n, n), dtype=int)
for t, p in zip(y_true, y_pred):
m[t, p] += 1 # rows true, columns predicted
return m
def per_class(M):
tp = np.diag(M).astype(float)
fp = M.sum(axis=0) - tp # column total minus the diagonal
fn = M.sum(axis=1) - tp # row total minus the diagonal
z = np.zeros_like(tp)
prec = np.divide(tp, tp + fp, out=z.copy(), where=(tp + fp) > 0)
rec = np.divide(tp, tp + fn, out=z.copy(), where=(tp + fn) > 0)
f1 = np.divide(2 * prec * rec, prec + rec, out=z.copy(), where=(prec + rec) > 0)
return prec, rec, f1, M.sum(axis=1)
M = confusion_matrix(y_true, y_pred, 3)
print(M)
prec, rec, f1, support = per_class(M)
for i, name in enumerate(labels):
print(f"{name:9} precision {prec[i]:.3f} recall {rec[i]:.3f} f1 {f1[i]:.3f} n {support[i]}")
print("macro ", round(float(f1.mean()), 3))
print("weighted", round(float((f1 * support).sum() / support.sum()), 3))
print("micro ", round(float(np.diag(M).sum() / M.sum()), 3), "(= accuracy)")
# [[ 0 5 0]
# [ 0 30 5]
# [ 0 0 60]]
# account precision 0.000 recall 0.000 f1 0.000 n 5
# billing precision 0.857 recall 0.857 f1 0.857 n 35
# bug precision 0.923 recall 1.000 f1 0.960 n 60
# macro 0.606
# weighted 0.876
# micro 0.9 (= accuracy)from sklearn.metrics import f1_score
f1_score(y_true, y_pred, average="macro")
f1_score(y_true, y_pred, average="weighted")Score the probabilities#
Accuracy only checks whether the top guess was right. If the probabilities are used, for pricing risk or deciding when to ask a person, score them directly. Log loss punishes confident wrong answers heavily; the Brier score is the mean squared error of the probabilities and is easier to interpret.
from sklearn.metrics import log_loss, brier_score_loss
log_loss(y_binary, p_positive) # lower is better; confident mistakes cost a lot
brier_score_loss(y_binary, p_positive) # 0 is perfect; always predicting 0.5 gives 0.25Neither number tells you the shape of the problem, and the shape is what you fix. For that, bin the predictions and compare what the model said against what actually happened in each bin. A well-calibrated model's two columns match.
import numpy as np
def calibration(y, p, bins=5):
edges = np.linspace(0, 1, bins + 1)
idx = np.clip(np.digitize(p, edges[1:-1]), 0, bins - 1)
for b in range(bins):
m = idx == b
if m.sum():
print(f"{edges[b]:.1f}-{edges[b+1]:.1f} n={int(m.sum()):5d}"
f" said {p[m].mean():.3f} happened {y[m].mean():.3f}")
rng = np.random.default_rng(0)
n = 4000
true_p = rng.uniform(0.05, 0.95, n)
y = (rng.random(n) < true_p).astype(int)
overconfident = true_p ** 2 / (true_p ** 2 + (1 - true_p) ** 2) # pushed to the extremes
calibration(y, overconfident)
# 0.0-0.2 n= 1243 said 0.067 happened 0.195
# 0.2-0.4 n= 574 said 0.296 happened 0.415
# 0.4-0.6 n= 428 said 0.501 happened 0.505
# 0.6-0.8 n= 512 said 0.703 happened 0.635
# 0.8-1.0 n= 1243 said 0.930 happened 0.808Ranking metrics cannot see this at all
The overconfident scores above are a monotone transform of the honest ones, so they order every pair identically and ROC AUC is 0.795 for both. Log loss separates them, 0.549 against 0.625. If a number from your model is multiplied by a pound amount anywhere downstream, AUC is not enough and calibration is the thing to check.
Per slice#
df = X_test.assign(y=y_test, pred=model.predict(X_test))
for group, part in df.groupby("country"):
print(group, len(part), f1_score(part.y, part.pred, average="macro"))Which slices, decided before you look: the ones where the population differs, the ones where a failure is expensive, and the ones that are small enough to be invisible in the average. In practice that usually means country or language, new users against established ones, mobile against desktop, and the most recent week against the rest. Anything with fewer than a few hundred examples in a slice has an error bar wider than most of the differences you will be tempted to act on, so compute the count alongside the score and say so.
Slice averages can also reverse the overall result. A model can be worse than another on every single segment and better overall, if the segments are different sizes and the two models see different mixes. When an aggregate and a per-slice story disagree, the per-slice one is usually the one that describes what users experience.
Where people go wrong
Reporting one averaged number for a multi-class model. The class that matters most is often the one it gets wrong.
If this comes up in an interview
When would you use macro F1 instead of accuracy?
When classes are imbalanced and each one matters. Accuracy is dominated by the common classes; macro F1 averages per-class F1 equally, so a model that ignores a rare class is penalised.
Then they push further
Your weighted F1 is 0.91 and macro F1 is 0.62. What does that tell you?
Common classes are handled well and at least one rare class badly. Look at per-class scores and the confusion matrix to find which, then consider more data for it, class weights, or whether the class definition is clear.
What gets marked up. Marked up for explaining the gap between averages and moving straight to per-class analysis.
Build this: On a multi-class dataset with imbalance, report accuracy, macro and weighted F1, log loss, the confusion matrix, and metrics on two slices. Write two sentences on what each number hid.
Watch it explained
This is one section of Model evaluation & metrics, 16 hours across 4 sections. Read it in full to get the whole topic in order.