When one class is rare
By the end: You can build a useful model when the interesting class is 1 percent of the data.
- Why accuracy fails here
- Resampling and class weights
- Threshold tuning
- What to actually optimise
New to imbalanced and messy data? Many problems worth solving have a rare answer: one transaction in a thousand is fraud, one part in five hundred is defective. If the rare case is one per cent of the data, a model that always says no is right ninety-nine per cent of the time and completely useless. This topic is that situation and the handling of it — choosing a metric that cannot be gamed this way, weighting the rare class more heavily, resampling, and moving the threshold at which the model commits to an answer. Why it matters
When the thing you care about is one percent of the data, a model that always says no is ninety-nine percent accurate and completely useless. This section is about the case where the interesting class is rare: why the usual metric lies loudest here, what resampling and class weights actually do, and why the most useful lever is often the threshold you apply afterwards rather than anything in training.
Base rate Accuracy of "always no" What actually changes
10% 90% almost nothing; use the normal tools
1% 99% stop reporting accuracy; move the threshold
0.1% 99.9% threshold and cost become the whole problem
0.01% 99.99% ranking, not classification: score and review the top k
1 in a million 99.9999% this is anomaly detection, not classificationThe setup#
Fraud, defects, disease, churn. The interesting class is rare, which means a model predicting the majority every time scores extremely well and is useless.
1000 transactions, 10 fraudulent
Always predict "not fraud":
accuracy 99%
fraud caught 0The arithmetic is worth doing once at a realistic ratio rather than a convenient one. Everything below uses a generated dataset of 200,000 rows with a positive rate of about 0.14 per cent, which is 700 to 1. That is an ordinary fraud or defect rate, and it is far enough into the table above that the usual advice starts to break down.
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, accuracy_score
X, y = make_classification(n_samples=200_000, n_features=20, n_informative=6,
weights=[0.999, 0.001], flip_y=0.001, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0)
print(ytr.sum(), len(ytr)) # 201 positives in 140,000 rows = 0.144%
print(1 - yte.mean()) # 0.99857 accuracy of a model that says no to everything
plain = LogisticRegression(max_iter=2000).fit(Xtr, ytr)
s = plain.predict_proba(Xte)[:, 1]
tn, fp, fn, tp = confusion_matrix(yte, s >= 0.5).ravel()
print(accuracy_score(yte, s >= 0.5), tp, fp, fn)
# 0.99858 tp 1 fp 0 fn 85That is the failure in full. The model is not broken and it is not untrained; it has correctly learned that predicting the majority is almost always right, because that is exactly what the loss function asked for. Accuracy went up. One case in 86 was caught. Reported as 99.86 per cent accurate, this would be signed off.
The options#
- Class weights: tell the model a mistake on the rare class costs more. Simplest, usually try first.
- Oversampling the minority, including synthetic methods like SMOTE. Risks overfitting to the few real examples.
- Undersampling the majority. Fast, and throws away data you paid for.
- Threshold tuning: leave the model alone and move the decision boundary. Frequently the largest win for the least effort.
model = LogisticRegression(class_weight="balanced").fit(Xtr, ytr)
probs = model.predict_proba(Xte)[:, 1]
preds = probs > 0.2 # not 0.5, chosen from the PR curveWeighting and resampling are usually presented as alternatives. They are close to the same operation: balanced class weights multiply the rare class's contribution to the loss by the inverse of its frequency, and duplicating each rare row that many times adds it to the loss that many times. Written out side by side and measured, they land in the same place.
import numpy as np
# (a) class weights: one argument
bal = LogisticRegression(max_iter=2000, class_weight="balanced").fit(Xtr, ytr)
# (b) random oversampling, by hand, on the training fold only
pos, neg = np.where(ytr == 1)[0], np.where(ytr == 0)[0]
up = np.random.default_rng(0).choice(pos, len(neg), replace=True) # draw with replacement
idx = np.concatenate([neg, up])
over = LogisticRegression(max_iter=2000).fit(Xtr[idx], ytr[idx])
s_bal = bal.predict_proba(Xte)[:, 1] # scores, kept for the comparison below
s_over = over.predict_proba(Xte)[:, 1]
# accuracy precision recall tp fp fn
# plain, threshold 0.5 0.99858 1.000 0.012 1 0 85
# class_weight=balanced 0.76190 0.005 0.756 65 14265 21
# oversampled 1:1 0.76205 0.005 0.756 65 14256 21Both moved recall from 0.012 to 0.756, which looks like the problem solved. It is not. They also produced 14,265 alerts to catch 65 real cases, so 99.5 per cent of everything sent to a human is wrong. Whether that is progress depends entirely on what an alert costs, which is a question neither method asked.
The threshold does the same job, better#
The plain model was never unable to find the positives. It ranked them fine and then applied a cut-off of 0.5, which is a number with no connection to anything. Move it, and the untouched model covers the whole range of behaviours the rebalanced ones offered.
for t in (0.5, 0.1, 0.01, 0.005, 0.002):
tn, fp, fn, tp = confusion_matrix(yte, s >= t).ravel()
print(t, tp, fp, fn)
# threshold precision recall tp fp fn
# 0.500 1.000 0.012 1 0 85
# 0.100 0.938 0.174 15 1 71
# 0.010 0.035 0.500 43 1185 43
# 0.005 0.015 0.605 52 3455 34
# 0.002 0.006 0.733 63 10182 23 <- compare with the rebalanced rowsSo the comparison is not close. At roughly the same recall the plain model with a moved threshold produces 10,182 false positives where the rebalanced ones produced 14,265, which is 29 per cent less work for the team reading the alerts. And the underlying ranking, which is what the threshold is slicing, was measurably damaged by rebalancing.
from sklearn.metrics import roc_auc_score, average_precision_score
for name, sc in [("plain", s), ("balanced", s_bal), ("oversampled", s_over)]:
print(name, round(roc_auc_score(yte, sc), 4), round(average_precision_score(yte, sc), 4))
# plain roc_auc 0.8451 average_precision 0.2332
# balanced roc_auc 0.8460 average_precision 0.1751
# oversampled roc_auc 0.8463 average_precision 0.1731That is the finding worth carrying away, because it is the opposite of the usual advice. Rebalancing changes where the model puts its decision boundary, which the threshold already controls for free, and in doing so it distorts the fit in a way that can make the ranking worse. Try the threshold first. Reach for weights when the model genuinely cannot separate the classes at any threshold, which the precision-recall curve will tell you.
Rebalancing destroys the probabilities
A model trained with balanced weights on this data predicts a mean probability of 0.357 where the true rate is 0.00143, overstating the risk by a factor of 250. The rank order survives; the numbers do not. If anything downstream multiplies the score by a transaction value, or shows a percentage to a person, rebalance and you are now lying by two orders of magnitude. Calibrate afterwards, or do not rebalance.
Resample training only, never validation
If you balance the validation set you are measuring on a distribution that does not exist, and the reported performance will not survive contact with production. Resample the training fold; evaluate on the real distribution.
Optimise the business quantity#
Precision and recall are proxies. What usually matters is a cost: a missed fraud costs the transaction value, a false positive costs a review. Expressing that directly gives you an unambiguous threshold rather than an argument about metrics.
def expected_cost(y_true, probs, threshold, miss_cost, review_cost):
preds = probs > threshold
misses = ((y_true == 1) & ~preds).sum()
reviews = preds.sum()
return misses * miss_cost + reviews * review_cost
# A missed case costs 500, a reviewed alert costs 8. Sweep and take the minimum.
grid = [0.001, 0.002, 0.005, 0.01, 0.05, 0.1, 0.5]
for t in grid:
print(t, expected_cost(yte, s, t, 500, 8))
best = min(grid, key=lambda t: expected_cost(yte, s, t, 500, 8))
print("cheapest threshold", best)
# 0.001 159216
# 0.002 93460
# 0.005 45056
# 0.01 31324
# 0.05 34972
# 0.1 35628
# 0.5 42508
# cheapest threshold 0.01Done properly that needs three splits, not two: train the model, choose the threshold on validation, and confirm on a test set that neither has touched. Choosing the threshold on the same data you report is the same overfitting as any other hyperparameter, and with a rare class it is worse, because the choice is being made from a handful of positives.
from sklearn.metrics import precision_recall_curve
Xtr, Xtmp, ytr, ytmp = train_test_split(X, y, test_size=0.4, stratify=y, random_state=0)
Xva, Xte, yva, yte = train_test_split(Xtmp, ytmp, test_size=0.5, stratify=ytmp, random_state=0)
m = LogisticRegression(max_iter=2000).fit(Xtr, ytr)
sva, ste = m.predict_proba(Xva)[:, 1], m.predict_proba(Xte)[:, 1]
MISS, REVIEW = 400, 5 # pounds per missed case, per alert reviewed
_, _, thr = precision_recall_curve(yva, sva)
best_t = min(thr, key=lambda t: MISS * ((sva < t) & (yva == 1)).sum() + REVIEW * (sva >= t).sum())
print(round(best_t, 5)) # 0.01656 chosen on validation only
# confirmed on the held-out test set:
# threshold 0.5 alerts 1 caught 1/57 missed 56 cost 22,405
# cost-chosen alerts 308 caught 26/57 missed 31 cost 13,940The two costs only have to be right relative to each other; scaling both changes nothing. That is useful in practice because nobody can tell you what a missed case costs, but almost everyone can tell you whether it is worth ten reviews or a thousand. Ask the question that way and you will get an answer.
At 1000 to 1, optimise the ranking#
Far enough down the table at the top of this section, the framing changes. Nobody is going to act on 14,000 alerts, so the model is not deciding anything; it is ordering a queue that a team will work down until they run out of day. The right question stops being where the threshold goes and becomes how much of the rare class sits in the top k, where k is a staffing number rather than a modelling one.
for k in (100, 500, 2000):
top = np.argsort(-ste)[:k]
print(k, round(yte[top].mean(), 3), round(yte[top].sum() / yte.sum(), 3))
# review top 100: precision 0.170 recall 0.298
# review top 500: precision 0.058 recall 0.509
# review top 2000: precision 0.018 recall 0.649Read those rows as an operational choice. One hundred reviews a day catches 30 per cent of cases and one alert in six is real, which a team will tolerate. Two thousand reviews catches 65 per cent and 98 per cent of the work is wasted, which no team will tolerate for long, and the usual outcome is that they stop reading the alerts entirely. The model that maximises recall is frequently the model that gets switched off.
Metric Use it when Why not otherwise
accuracy never, with a rare class the majority class answers it
ROC AUC comparing models during development flattered by the huge TN count
average precision comparing models, rare class respects the base rate
precision at k a fixed review capacity matches what the team does
recall at fixed alerts a fixed review capacity the same trade, stated the other way
expected cost the costs are roughly known turns the argument into arithmetic
lift over base rate explaining value to a non-specialist "120 times better than random"Not enough positives to measure
With 201 positives in the training set, every number here has a wide confidence interval, and a difference of two percentage points of recall is eight cases and probably noise. Bootstrap the test set to get an interval before believing any comparison, use stratified cross-validation rather than a single split, and accept that below roughly fifty positives in the evaluation set you cannot distinguish two decent models at all. Collecting more labelled positives is then a better use of a week than any modelling change.
Where people go wrong
Balancing the test set. Performance then looks strong on a distribution that does not exist, and drops immediately in production.
If this comes up in an interview
Your classes are 99 to 1. What do you do?
Stop using accuracy, and use precision, recall and PR AUC. Try class weights first, tune the decision threshold from the precision-recall curve rather than leaving it at 0.5, and evaluate on the real distribution rather than a rebalanced one.
Should you oversample before or after splitting?
After, and only the training fold. Oversampling before splitting puts copies of the same example in both train and test, which inflates the score. The validation set must keep the real class balance.
How do you pick the decision threshold?
From the cost of each error type. Compute expected cost across thresholds using the real cost of a miss and of a false positive, and pick the minimum. That turns a metric argument into an arithmetic answer.
Build this: Take an imbalanced dataset and compare class weights, SMOTE and threshold tuning. Report precision, recall and expected cost for each, and note which gave the most improvement per unit of effort.
Watch it explained
This is one section of Imbalanced and messy data, 9 hours across 2 sections. Read it in full to get the whole topic in order.