Memorising instead of learning

A model that does well on the data it was trained on has proved nothing. This section is about the gap between that score and how it performs on anything else, what causes the gap to open, and how to see it happening in a learning curve rather than discovering it after launch. The fixes are a short list and they have an order, which is most of what makes this practical rather than theoretical.

The symptom#

The model performs far better on training data than on data it has not seen. It has memorised the examples, including their noise, rather than learning the pattern.

train accuracy 0.99    test accuracy 0.71     overfitting
train accuracy 0.72    test accuracy 0.70     underfitting
train accuracy 0.91    test accuracy 0.88     about right

That table is the summary. What is worth seeing once, properly, is the gap opening while you watch, because it does not open in the way most descriptions imply. Here is a gradient boosting model fitted to 700 rows of a problem where 15 per cent of the labels have been flipped, so there is a real pattern and real noise sitting on top of it. Every one of the 500 trees is scored on the training rows and on 3,000 rows the model never sees.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import accuracy_score, log_loss

X, y = make_classification(n_samples=10_000, n_features=25, n_informative=6,
                           n_redundant=0, flip_y=0.15, random_state=0)
Xva, yva = X[7000:], y[7000:]          # 3000 rows, fixed, never trained on
Xtr, ytr = X[:700], y[:700]            # the training set we start with

m = GradientBoostingClassifier(n_estimators=500, max_depth=4, random_state=0).fit(Xtr, ytr)
tr = list(m.staged_predict_proba(Xtr))
va = list(m.staged_predict_proba(Xva))

print("trees  train acc  val acc    gap   val log loss")
for i in (9, 24, 49, 99, 199, 499):
    a = accuracy_score(ytr, tr[i].argmax(1))
    b = accuracy_score(yva, va[i].argmax(1))
    print(f"{i+1:5d}  {a:9.3f}  {b:7.3f}  {a-b:+.3f}   {log_loss(yva, va[i]):8.3f}")
best = int(np.argmin([log_loss(yva, p) for p in va]))
print("best validation log loss at tree", best + 1)

# trees  train acc  val acc    gap   val log loss
#    10      0.873    0.774  +0.099      0.521
#    25      0.913    0.787  +0.126      0.478
#    50      0.961    0.785  +0.176      0.476
#   100      0.999    0.783  +0.216      0.488
#   200      1.000    0.781  +0.219      0.528
#   500      1.000    0.788  +0.212      0.717
# best validation log loss at tree 37

Read the last column before the others. Validation accuracy is almost flat across the whole run, so a team tracking accuracy alone would see nothing wrong and ship the 500-tree model. Validation log loss, which scores how confident the model was rather than only whether it was right, bottoms out at tree 37 and then rises to 0.717 — half again as bad as where it started. That is what overfitting does first. The model does not begin getting more answers wrong; it begins being more certain about the ones it gets wrong, and that is the part that shows up as a badly calibrated probability in production long before it shows up as a drop in accuracy.

The other number to take from the table is 0.212. Train minus validation is the gap, and it is the quantity you monitor. A gap that grows while validation stands still is the signature; a gap near zero with both numbers poor is the opposite problem and needs the opposite treatment.

Bias and variance#

These two words get defined and then never used. They are worth making concrete, because the whole of the rest of this section is a decision about which of the two you have. Take a known function, sine over the unit interval, add noise of a known size, and fit polynomials of increasing degree. Because the true function is known, the error of a fitted model can be split into exactly three parts: how far the average fit sits from the truth (bias), how much the fit moves when the training sample changes (variance), and the noise, which nothing can remove.

rng = np.random.default_rng(0)
NOISE, GRID = 0.3, np.linspace(0, 1, 100)
truth = np.sin(2 * np.pi * GRID)

print("degree    bias^2  variance   noise     sum   test MSE")
for deg in (0, 1, 3, 5, 7, 9):
    preds = []
    for _ in range(300):                      # 300 independent training sets of 30 points
        x = rng.random(30)
        yy = np.sin(2 * np.pi * x) + rng.normal(0, NOISE, 30)
        preds.append(np.polyval(np.polyfit(x, yy, deg), GRID))
    P = np.array(preds)
    bias2 = float(((P.mean(0) - truth) ** 2).mean())
    var = float(P.var(0).mean())
    test = truth + rng.normal(0, NOISE, (300, 100))   # fresh noisy labels, as a check
    mse = float(((P - test) ** 2).mean())
    print(f"{deg:6d} {bias2:9.3f} {var:9.3f} {NOISE**2:7.3f} {bias2+var+NOISE**2:7.3f} {mse:10.3f}")

# degree    bias^2  variance   noise     sum   test MSE
#      0     0.495     0.023   0.090   0.608      0.607
#      1     0.204     0.021   0.090   0.315      0.314
#      3     0.005     0.017   0.090   0.112      0.113
#      5     0.000     0.074   0.090   0.165      0.164
#      7     0.004     0.573   0.090   0.667      0.669
#      9     0.022     6.499   0.090   6.611      6.607

Bias falls to nothing by degree 3 and stays there — a cubic can already trace a sine closely enough. Variance is 0.017 at degree 3 and 6.499 at degree 9, a factor of nearly four hundred, from thirty training points that differ only in which thirty they are. The best total error is at degree 3, and it is 0.112 against a noise floor of 0.090, so 80 per cent of the remaining error is irreducible. Nothing in this section can beat 0.090, and a project that has not worked out its own equivalent of that number will keep spending on the part that cannot move.

• High bias: the model is too simple to capture the pattern. Both scores are poor. Underfitting.
• High variance: the model is complex enough to fit noise. Training score far exceeds test. Overfitting.
• More capacity reduces bias and increases variance. Finding the balance for your data volume is the bias-variance tradeoff.
• Above degree 5 here, bias is flat and total error is rising: every unit of extra capacity is buying variance and nothing else. That is the exact condition under which reducing capacity helps and nothing else will.

Learning curves#

Plot training and validation performance against training set size, or against epochs. The shape tells you which problem you have and what will help.

Curves converged, both poor          more data will not help
                                      you need a better model

Large gap, validation flat           more data will help
                                      or reduce complexity

Validation rising then falling       stop earlier

That is the rule. Here are both cases measured, so you can see what each actually looks like before you have to read one under pressure. Case A is a random forest on a thirty-column problem; case B is logistic regression on two interleaved crescents, a shape no straight line can separate.

from sklearn.datasets import make_moons
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import learning_curve

Xa, ya = make_classification(n_samples=20000, n_features=30, n_informative=20,
                             n_redundant=0, flip_y=0.05, class_sep=0.8, random_state=0)
Xb, yb = make_moons(n_samples=20000, noise=0.3, random_state=0)

for name, model, XX, yy in [
    ("A  random forest, 30 columns", RandomForestClassifier(n_estimators=100, random_state=0), Xa, ya),
    ("B  logistic regression, two moons", LogisticRegression(), Xb, yb)]:
    n, t, v = learning_curve(model, XX, yy, train_sizes=[200, 500, 1500, 5000, 15000],
                             cv=4, scoring="accuracy", shuffle=True, random_state=0)
    print(name)
    for i, s in enumerate(n):
        print(f"   n={s:6d}   train {t[i].mean():.3f}   val {v[i].mean():.3f}   gap {t[i].mean()-v[i].mean():+.3f}")

# A  random forest, 30 columns
#    n=   200   train 1.000   val 0.697   gap +0.303
#    n=   500   train 1.000   val 0.796   gap +0.204
#    n=  1500   train 1.000   val 0.863   gap +0.137
#    n=  5000   train 1.000   val 0.910   gap +0.090
#    n= 15000   train 1.000   val 0.934   gap +0.066
# B  logistic regression, two moons
#    n=   200   train 0.867   val 0.845   gap +0.022
#    n=   500   train 0.857   val 0.848   gap +0.009
#    n=  1500   train 0.851   val 0.850   gap +0.001
#    n=  5000   train 0.851   val 0.849   gap +0.002
#    n= 15000   train 0.850   val 0.849   gap +0.001

Case A: the gap starts at 0.303 and is still closing at 15,000 rows, and validation has climbed from 0.697 to 0.934 without flattening. Every additional row is still buying accuracy, so the answer is more data, and the curve even tells you roughly how much — the last tripling bought 2.4 points, so the next one will buy less than that and you can decide whether it is worth the labelling cost before spending it.

Case B: the gap is 0.022 at two hundred rows and 0.001 at fifteen thousand. The model is not memorising anything; it has extracted everything it is capable of extracting and both curves have converged at 0.850. Seventy-five times more data moved validation accuracy by four thousandths. More data is not the fix and neither is regularisation, which only ever removes capacity from a model that does not have enough. A random forest on the same rows reaches 0.904, so the fix here is a model that can bend.

The levers#

1. 1.More data. Always the best fix when available, and usually the least available.
2. 2.Early stopping. Free, and stops training at the point validation was best.
3. 3.Regularisation. L2 penalises large weights; L1 drives some to zero and selects features.
4. 4.Dropout, in neural networks. Randomly ignore units during training so the model cannot rely on any one.
5. 5.Reduce capacity. Fewer parameters, shallower trees, fewer features.
6. 6.Data augmentation, where the domain allows it.

The order is not arbitrary: it runs from the lever that raises the ceiling to the levers that only lower the gap. Applied to the overfitting model from the top of this section, four of them do something quite different from what the list implies.

import time

def score(model, Xt=Xtr, yt=ytr):
    t0 = time.perf_counter()
    model.fit(Xt, yt)
    return (accuracy_score(yt, model.predict(Xt)),
            accuracy_score(yva, model.predict(Xva)), time.perf_counter() - t0)

print("lever                          train    val     gap   fit s")
for label, model in [
    ("nothing (500 trees)", GradientBoostingClassifier(n_estimators=500, max_depth=4, random_state=0)),
    ("early stopping (37 trees)", GradientBoostingClassifier(n_estimators=37, max_depth=4, random_state=0)),
    ("less capacity (depth 2)", GradientBoostingClassifier(n_estimators=500, max_depth=2, random_state=0)),
    ("slower, subsampled", GradientBoostingClassifier(n_estimators=500, max_depth=4,
                                                      subsample=0.5, learning_rate=0.02, random_state=0)),
]:
    a, b, s = score(model)
    print(f"{label:30s} {a:.3f}  {b:.3f}  {a-b:+.3f}  {s:5.1f}")
a, b, s = score(GradientBoostingClassifier(n_estimators=500, max_depth=4, random_state=0), X[:7000], y[:7000])
print(f"{'10x the data (7000 rows)':30s} {a:.3f}  {b:.3f}  {a-b:+.3f}  {s:5.1f}")

# lever                          train    val     gap   fit s
# nothing (500 trees)            1.000  0.788  +0.212    1.8
# early stopping (37 trees)      0.946  0.786  +0.159    0.1
# less capacity (depth 2)        0.999  0.765  +0.234    0.9
# slower, subsampled             1.000  0.790  +0.210    0.9
# 10x the data (7000 rows)       0.974  0.842  +0.132   22.2

Only one lever moved validation accuracy, and it is the one at the top of the list: ten times the data bought 5.4 points, from 0.788 to 0.842, for thirteen times the fit time. Early stopping gave back 0.002 of accuracy and took eighteen times less time to train — and, as the first table showed, it halved the log loss, which is the reason to do it. Slowing the learning rate and subsampling gained 0.002, inside the noise. Reducing depth to 2 lost 2.3 points: the model was not too complex for the problem, it was too complex for 700 rows, and taking capacity away made it worse at both.

That last row is the one worth remembering, because it is the standard mistake. Seeing a gap of 0.212 and reaching for less capacity treats the gap as the thing to minimise. The gap is a symptom. What you are minimising is validation error, and a change that closes the gap by making the training score worse has usually done nothing for you. Check both numbers after every lever, and keep the one that moved the second.