Regularisation and validation, hands on
By the end: You can apply the main regularisers, validate with the right kind of split, and tell which one your problem needs.
- L2 and L1 penalties
- Early stopping
- Dropout and augmentation
- Cross-validation done properly
- Splits for grouped and time-ordered data
New to overfitting & regularisation? A model can become very good at the examples it was trained on without learning anything that transfers. It is the student who memorised last year's exam paper: flawless on those questions, lost the moment the wording changes. Overfitting is that failure, and regularisation is the family of techniques that hold it back by deliberately limiting how closely a model is allowed to fit the data in front of it. Why it matters
Two questions decide whether a model generalises: what stops it memorising the training data, and whether the validation score you are reading is honest. This section covers both. First the regularisers, an L2 or L1 penalty, early stopping, dropout and augmentation, and then the split, which is where most inflated scores come from: the same customer on both sides, a scaler fitted before the split, or time-ordered rows that were shuffled. A validation number produced by a careless split is worse than no number, because you act on it.
Penalise large weights#
An L2 penalty adds the sum of squared weights to the loss, so the model only uses large weights when they clearly pay for themselves. L1 adds the sum of absolute weights, which pushes many weights to exactly zero and so also selects features.
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
# 1,400 rows, 200 columns, 15 of which carry any signal at all.
X, y = make_classification(n_samples=1400, n_features=200, n_informative=15,
n_redundant=0, flip_y=0.1, random_state=0)
for C in [0.01, 0.1, 1, 10, 100]: # C is the inverse of penalty strength
model = LogisticRegression(C=C, max_iter=2000)
scores = cross_val_score(model, X, y, cv=5, scoring="roc_auc")
print(f"C={C:<6} AUC {scores.mean():.3f} +/- {scores.std():.3f}")
# C=0.01 AUC 0.764 +/- 0.019
# C=0.1 AUC 0.740 +/- 0.018
# C=1 AUC 0.734 +/- 0.019
# C=10 AUC 0.733 +/- 0.019
# C=100 AUC 0.733 +/- 0.019Extending it downwards is worth doing properly, because it reveals the thing the two penalties do differently. Below is the same data split into 400 training rows and a thousand held out, swept from a penalty so strong the model can barely move to one so weak it is effectively absent, for both L2 and L1, with the number of columns the fitted model actually uses.
import numpy as np
from sklearn.metrics import roc_auc_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
Xtr, ytr, Xva, yva = X[:400], y[:400], X[400:], y[400:]
def fit(**kw):
p = make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000, **kw)).fit(Xtr, ytr)
return (roc_auc_score(ytr, p.predict_proba(Xtr)[:, 1]),
roc_auc_score(yva, p.predict_proba(Xva)[:, 1]),
int((np.abs(p[-1].coef_) > 1e-6).sum()))
print("penalty C train AUC val AUC gap columns used")
for C in (1e-5, 1e-3, 1e-2, 1e-1, 1, 10):
t, v, nz = fit(C=C)
print(f"L2 {C:8} {t:9.3f} {v:8.3f} {t-v:+8.3f} {nz:9d}")
for C in (1e-2, 1e-1, 1, 10):
t, v, nz = fit(C=C, penalty="l1", solver="liblinear")
print(f"L1 {C:8} {t:9.3f} {v:8.3f} {t-v:+8.3f} {nz:9d}")
# penalty C train AUC val AUC gap columns used
# L2 1e-05 0.900 0.648 +0.252 200
# L2 0.001 0.911 0.647 +0.264 200
# L2 0.01 0.941 0.644 +0.297 200
# L2 0.1 0.974 0.637 +0.337 200
# L2 1 0.998 0.628 +0.370 200
# L2 10 1.000 0.623 +0.377 200
# L1 0.01 0.500 0.500 +0.000 0
# L1 0.1 0.876 0.728 +0.148 54
# L1 1 0.991 0.646 +0.345 167
# L1 10 1.000 0.632 +0.368 179L2 flattens out at 0.648 and cannot be pushed past it. Shrinking every weight towards zero still leaves 185 noise columns contributing a small amount each, and a small amount each, across 185 columns, is most of the model. L1 at the right strength deletes them outright and reaches 0.728. The top row of the L1 block is the other half of the lesson: at C=0.01 the penalty deletes everything, the model has no columns left and scores exactly 0.500, which is a coin flip. Both failure modes are a badly chosen strength rather than a badly chosen penalty, which is why the strength is swept rather than picked.
The rule that follows is short. When you have more columns than rows and expect most of them to be junk, L1 is doing a different job from L2 and usually the job you want. When the columns are all plausibly relevant and correlated with each other, L2 is better, because L1 picks one of a correlated pair arbitrarily and drops the other, which makes the fitted model unstable across refits even when its accuracy is fine.
Stop before it memorises#
For models trained in steps, such as neural networks and gradient boosting, validation loss usually falls, flattens, then rises as the model starts memorising. Early stopping keeps the weights from the best validation point. It is the cheapest regulariser there is.
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import log_loss
model = GradientBoostingClassifier(n_estimators=1, max_depth=3,
warm_start=True, random_state=0)
best, patience, bad, kept = float("inf"), 5, 0, 0
for epoch in range(1, 201):
model.set_params(n_estimators=epoch)
model.fit(X[:400], y[:400]) # warm_start adds one more tree
val = log_loss(y[400:], model.predict_proba(X[400:]))
if val < best:
best, bad, kept = val, 0, epoch # save(model) belongs here
else:
bad += 1
if bad >= patience:
break
print(f"stopped at epoch {epoch}, keeping epoch {kept}, val log loss {best:.4f}")
# stopped at epoch 22, keeping epoch 17, val log loss 0.6071Dropout and augmentation#
Dropout randomly zeroes activations during training, so the network cannot rely on any single path. Augmentation creates new training examples by transforming existing ones: flips and crops for images, noise for audio. Augmentation is often the strongest regulariser for images, because it adds real information about what should not change the label.
Cross-validation, and its traps#
K-fold cross-validation trains on k minus 1 parts and validates on the remaining one, rotating, so every example is used for validation once. It gives a mean and a spread. The spread matters: if two settings differ by less than it, you cannot tell them apart.
That sentence is easy to nod at and hard to feel, so here it is as a result. Two forest depths are compared on the same 600 rows, eight times over. Nothing changes between the eight runs except the random seed that decides where the fold boundaries fall.
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import KFold
Xs, ys = make_classification(n_samples=600, n_features=20, n_informative=8,
flip_y=0.15, random_state=0)
print("seed depth 6 depth 10 winner")
for seed in range(8):
cv = KFold(5, shuffle=True, random_state=seed)
a = cross_val_score(RandomForestClassifier(max_depth=6, n_estimators=200, random_state=0),
Xs, ys, cv=cv, scoring="roc_auc").mean()
b = cross_val_score(RandomForestClassifier(max_depth=10, n_estimators=200, random_state=0),
Xs, ys, cv=cv, scoring="roc_auc").mean()
print(f"{seed:4d} {a:8.4f} {b:9.4f} depth {6 if a > b else 10}")
# seed depth 6 depth 10 winner
# 0 0.9218 0.9194 depth 6
# 1 0.9238 0.9220 depth 6
# 2 0.9242 0.9250 depth 10
# 3 0.9251 0.9219 depth 6
# 4 0.9271 0.9279 depth 10
# 5 0.9251 0.9235 depth 6
# 6 0.9267 0.9260 depth 6
# 7 0.9242 0.9253 depth 10Depth 6 wins five times and depth 10 wins three, and the margins are in the third decimal place. A tuning run that reported the seed-0 result would say depth 6 by 0.0024 and a run that happened to use seed 4 would say depth 10 by 0.0008, and both would be presented as a finding. The honest summary of this experiment is that the two settings are indistinguishable on 600 rows. If you need to tell them apart, repeat the cross-validation with several seeds and compare the spread of the means to the gap between them, or get more data — those are the only two moves, and picking the winner of one split is not one of them.
- Classes are imbalanced: use stratified folds, so each fold keeps the class ratio.
- Several rows belong to one patient, user or document: use group folds, so one group never appears on both sides. Otherwise the model is tested on people it has already seen.
- Rows are ordered in time: train on the past, validate on the future, never shuffle.
- Any preprocessing that learns from data, such as scaling or target encoding, must be fitted inside each fold. A sklearn Pipeline does this for you.
import numpy as np
from sklearn.model_selection import GroupKFold, TimeSeriesSplit
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
# patient_ids is the id column of your own data, one entry per row.
patient_ids = np.repeat(np.arange(280), 5) # 280 patients, 5 visits each
pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))
cross_val_score(pipe, X, y, cv=GroupKFold(n_splits=5), groups=patient_ids)
cross_val_score(pipe, X, y, cv=TimeSeriesSplit(n_splits=5))The second item on that list is the one that costs the most and is spotted the least, so it is worth putting a number on. Two hundred patients, eight visits each, sixteen hundred rows. Every visit is drawn from its patient's underlying profile plus a little noise, and the label belongs to the patient rather than to the visit — which is how patient data, user data and document data are actually shaped.
rng = np.random.default_rng(0)
n_patients, per_patient, d = 200, 8, 12
profile = rng.normal(size=(n_patients, d)) # one latent profile per patient
label = (profile[:, 0] + rng.normal(0, 1.0, n_patients) > 0).astype(int)
groups = np.repeat(np.arange(n_patients), per_patient)
Xg = profile[groups] + rng.normal(0, 0.4, (n_patients * per_patient, d))
yg = label[groups] # every visit shares its patient's label
m = RandomForestClassifier(n_estimators=200, random_state=0)
a = cross_val_score(m, Xg, yg, cv=KFold(5, shuffle=True, random_state=0), scoring="roc_auc")
b = cross_val_score(m, Xg, yg, cv=GroupKFold(5), groups=groups, scoring="roc_auc")
print(f"shuffled k-fold AUC {a.mean():.3f} +/- {a.std():.3f}")
print(f"group k-fold AUC {b.mean():.3f} +/- {b.std():.3f}")
# shuffled k-fold AUC 0.947 +/- 0.016
# group k-fold AUC 0.735 +/- 0.044Twenty-one points of AUC, and the wrong number is the one that looks better and has the tighter spread. With shuffled folds the model does not need to generalise to a new patient; for almost every validation row there are seven near-copies of it in training, so it is being asked to recognise people it has already met. Group folds ask the question production will ask, which is whether the model works on a patient it has never seen, and the answer is 0.735.
Notice the spread as well as the mean. Group k-fold reports plus or minus 0.044 against 0.016, nearly three times wider, because there are only 200 independent units in this dataset rather than 1,600, and the honest error bar reflects that. An inflated score usually arrives with an inflated confidence, which is why a suspiciously tight cross-validation spread on entity-shaped data is worth checking before the mean is.
Data leakage#
Every trap above is one species of the same animal. Data leakage is information reaching the model that would not be available at prediction time, so the offline score measures something the production system will never get to do. It is the most common reason a model that validated at 0.95 serves at 0.70, and it is worth seeing rather than being warned about, because the warning never feels urgent enough.
import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(0)
X = rng.normal(size=(400, 800)) # 800 columns of pure noise
y = rng.integers(0, 2, 400) # a coin flip: nothing is predictable
# WRONG: pick the columns that correlate with y using all the data,
# then cross-validate on the survivors. The selection has already seen
# every validation fold, so the folds are no longer held out.
corr = np.array([abs(np.corrcoef(X[:, j], y)[0, 1]) for j in range(800)])
best = np.argsort(-corr)[:20]
print(round(cross_val_score(LogisticRegression(), X[:, best], y, cv=5).mean(), 3))
# RIGHT: the selection is a step in the pipeline, so it is refitted on
# the training part of each fold and never sees the validation part.
pipe = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression())
print(round(cross_val_score(pipe, X, y, cv=5).mean(), 3))
# 0.685 <- on data with no signal whatsoever
# 0.503 <- the truthThe question that finds it
For every feature, ask when its value becomes known. If the answer is 'after the thing I am predicting', it leaks — a refund flag when predicting fraud, a discharge code when predicting length of stay, a support-ticket count when predicting churn. This one question catches more leakage than any statistical check, and it is the follow-up interviewers reach for when a candidate says only 'the model overfit'.
Where people go wrong
Scaling the whole dataset before splitting. The validation folds have influenced the training data, and every score is slightly optimistic.
If this comes up in an interview
Your model scores 0.95 AUC in cross-validation and 0.70 in production. What might have happened?
Most often leakage in the validation setup: the same entity on both sides of the split, preprocessing fitted on all data, a feature that is only known after the outcome, or shuffling time-ordered data. Otherwise, production data differs from the training data.
Then they push further
How would you check for the same entity on both sides?
Look for an id such as user, patient or device that repeats across rows. Rerun cross-validation with group folds on that id. A large drop means the earlier score was inflated.
And for a feature that leaks the answer?
Check feature importance for anything suspiciously strong, and for each top feature ask when it becomes known. If it is only recorded after the outcome, it cannot be used.
What gets marked up. Strong answers go straight to the validation setup and name group, time and preprocessing leakage. 'The model overfit' without a mechanism is marked down.
What is the difference between L1 and L2 regularisation?
L2 penalises squared weights and shrinks them all smoothly. L1 penalises absolute weights and drives many to exactly zero, which also performs feature selection.
Build this: Take a dataset with repeated entities, such as several rows per customer. Compare scores from shuffled k-fold and group k-fold, with preprocessing inside a Pipeline. Then sweep one regularisation strength and plot the mean and spread.
Watch it explained
This is one section of Overfitting & regularisation, 10 hours across 2 sections. Read it in full to get the whole topic in order.