Metrics and thresholds, worked through
By the end: You can read a precision-recall trade-off, choose a threshold from costs, and check whether the model's probabilities can be trusted.
- A confusion matrix with real numbers
- Precision-recall curves
- Why ROC AUC flatters rare-event models
- Choosing a threshold from costs
- Calibration
New to imbalanced and messy data? Many problems worth solving have a rare answer: one transaction in a thousand is fraud, one part in five hundred is defective. If the rare case is one per cent of the data, a model that always says no is right ninety-nine per cent of the time and completely useless. This topic is that situation and the handling of it — choosing a metric that cannot be gamed this way, weighting the rare class more heavily, resampling, and moving the threshold at which the model commits to an answer. Why it matters
A classifier produces a score; the threshold turns it into a decision, and choosing it is a business question rather than a modelling one. This section works through it with real numbers on a rare-event problem: reading a confusion matrix, why ROC AUC flatters a model when the positive class is 0.2% of the data while precision-recall describes what the team reviewing alerts will actually experience, and how to pick the threshold from the cost of a miss against the cost of a review. It ends on calibration, which you need the moment anything downstream treats the score as a real probability.
The numbers#
100,000 transactions, 200 are fraud (0.2%). Threshold 0.5:
predicted fraud predicted fine
actual fraud 150 50
actual fine 850 98,950
precision = 150 / (150 + 850) = 15% (of alerts, how many are fraud)
recall = 150 / (150 + 50) = 75% (of fraud, how much we caught)
accuracy = 99,100 / 100,000 = 99.1% (a model saying 'fine' always gets 99.8%)Read the two definitions carefully, because the difference between them is the whole section. Precision divides by the number of alerts, so it answers the reviewer's question: if this fires, how often is it real. Recall divides by the number of real cases, so it answers the risk question: of everything that happened, how much did we see. They are computed from the same confusion matrix and they move in opposite directions as the threshold slides, which is why quoting one without the other says nothing.
Every threshold is a different model#
The classifier outputs a score. The threshold turns it into a decision. Raising it gives fewer alerts, higher precision and lower recall. A precision-recall curve shows every trade-off at once, and the area under it summarises the model across thresholds.
from sklearn.metrics import precision_recall_curve, average_precision_score
precision, recall, thresholds = precision_recall_curve(y_val, scores)
print("PR AUC (average precision):", average_precision_score(y_val, scores))One detail about the curve that catches people: precision is not monotonic along it. Raising the threshold usually raises precision, but it can drop when the next few cases excluded happen to be real, and at the extreme right-hand end the curve is computed from a handful of examples and is mostly noise. Do not choose an operating point from the ragged end of a PR curve; it will not reproduce.
A concrete curve, base rate 0.14%, 60,000 rows:
threshold precision recall what it means operationally
0.0508 0.333 0.246 1 alert in 3 is real, you catch a quarter
0.0200 0.107 0.404 1 in 9 is real, you catch 40%
0.0100 0.041 0.544 1 in 24 is real, you catch half
0.0050 0.016 0.649 1 in 62 is real
0.0020 0.006 0.789 1 in 164 is real, you catch four fifths
average precision 0.221 a random model would score 0.0014Average precision has a property that makes it the right summary here: its random baseline is the base rate itself. So 0.221 against a base rate of 0.0014 means the model is roughly 150 times better than guessing, and that ratio stays meaningful as the base rate changes. ROC AUC has a fixed random baseline of 0.5 regardless of imbalance, which is precisely why it cannot tell you how hard your problem is.
Why ROC AUC can mislead here#
ROC curves use the false positive rate, which divides by the huge number of legitimate transactions. 850 false alarms out of 99,800 is a false positive rate under 1%, which looks excellent, while analysts are drowning in alerts that are 85% wrong. For rare events, precision-recall tells you what the team will experience.
The same three operating points, described both ways, is the clearest version of this. Nothing differs but the denominator.
threshold false positive rate precision recall
0.010 0.0198 0.0350 0.500
0.005 0.0577 0.0148 0.605
0.002 0.1699 0.0061 0.733
At threshold 0.01: "a 2% false positive rate" and "96.5% of alerts are wrong"
are the same sentence about the same model.print(round(roc_auc_score(yte, s), 4)) # 0.8451 random = 0.5
print(round(average_precision_score(yte, s), 4)) # 0.2332 random = 0.00143Pick the threshold from money#
COST_MISSED_FRAUD = 400 # average loss per missed fraud
COST_REVIEW = 5 # analyst time per alert
best = min(
thresholds,
key=lambda t: COST_MISSED_FRAUD * ((scores < t) & (y_val == 1)).sum()
+ COST_REVIEW * (scores >= t).sum(),
)Can you trust the probabilities?#
A model is calibrated if, among cases it scores at 0.3, about 30% are positive. Resampling and class weights, common fixes for imbalance, distort probabilities. If anything downstream uses the number as a probability, such as expected loss, calibrate on held-out data and check with a reliability curve.
from sklearn.calibration import CalibratedClassifierCV, calibration_curve
calibrated = CalibratedClassifierCV(model, method="isotonic", cv=5).fit(X_train, y_train)
frac_pos, mean_pred = calibration_curve(y_val, calibrated.predict_proba(X_val)[:, 1], n_bins=10)The quickest check is one line before any curve: compare the mean predicted probability with the observed rate. A calibrated model matches, because the average of its predictions has to equal the average outcome. Measured on the same 0.14 per cent dataset:
for name, sc in [("plain", s), ("balanced", s_bal)]:
print(name, round(sc.mean(), 5), round(yte.mean(), 5))
# plain mean predicted 0.00147 observed 0.00143 ratio 1.0x
# balanced mean predicted 0.35665 observed 0.00143 ratio 248.8xA reliability curve is the same idea per bucket: sort by score, cut into groups, and compare what was predicted with what happened. Quantile bins rather than equal-width bins, because with a rare class every equal-width bin above 0.1 is empty.
Plain model, five quantile bins:
predicted observed
0.00005 0.00008
0.00019 0.00058
0.00047 0.00050
0.00110 0.00067
0.00555 0.00533 close on every bin: this model can be trusted as a probabilityIsotonic regression repairs the scale, and it is worth knowing exactly what it does and does not repair. Applied to the rebalanced model it brings the mean prediction back from 0.357 to 0.00140 against a true rate of 0.00143, which is the fix working. It does not recover the ranking that rebalancing cost.
cal = CalibratedClassifierCV(LogisticRegression(max_iter=2000, class_weight="balanced"),
method="isotonic", cv=3).fit(Xtr, ytr)
sc = cal.predict_proba(Xte)[:, 1]
print(round(sc.mean(), 5), round(yte.mean(), 5)) # 0.00140 0.00143 fixed
print(round(average_precision_score(yte, s_bal), 4),
round(average_precision_score(yte, sc), 4)) # 0.1751 -> 0.1680 not fixedIsotonic or Platt
Isotonic fits an arbitrary increasing step function and is the better choice with enough data, roughly a thousand positives. Below that it overfits, producing flat steps where a whole range of scores maps to one probability. Platt scaling fits a one-parameter sigmoid instead, which cannot overfit in the same way and is the right default when positives are scarce, which on a rare-event problem they always are. Fit either on held-out data, never on the training set.
Where people go wrong
Using the default 0.5 threshold. For a rare class it is almost never the right one, because nothing about 0.5 relates to your costs.
If this comes up in an interview
Your fraud model has 0.97 ROC AUC. The fraud team says it is useless. Who is right?
Possibly both. With fraud at 0.2%, even a low false positive rate produces far more false alerts than real fraud, so precision at the operating threshold can be very low. I would look at the precision-recall curve and at precision and alert volume at the threshold actually in use.
Then they push further
How would you choose the threshold?
From costs: the average loss from a missed fraud against the cost of reviewing an alert, plus any capacity limit on the team. Pick the threshold that minimises expected cost on validation data, then confirm on held-out data.
The team can only review 500 alerts a day. Does that change anything?
Yes. Then the question becomes which 500 to review, so rank by score and take the top 500. The metric that matters is precision in the top 500, and improving the model means pushing more fraud into that set.
What gets marked up. Strong answers explain why ROC AUC flatters rare-event models and move to precision at the operating point and costs. Treating AUC as the final answer is marked down.
What does it mean for a classifier to be calibrated, and when do you need it?
Its scores match observed frequencies: cases scored 0.3 are positive about 30% of the time. You need it whenever the score is used as a probability, such as in expected cost calculations or when showing risk to people.
Build this: Take an imbalanced dataset, train a model with class weights, plot its precision-recall curve, choose a threshold from assumed costs, and draw a reliability curve before and after isotonic calibration.
Watch it explained
This is one section of Imbalanced and messy data, 9 hours across 2 sections. Read it in full to get the whole topic in order.