Hyperparameter tuning

There are dozens of settings you could adjust and almost all of them barely matter. This section is about doing it in order: the learning rate dominates everything else, random sampling beats an exhaustive grid for the same budget, and the more expensive each run is the more it is worth being clever about which to run next. The discipline that matters most is never tuning against the set you report on.

Two words before anything else, because they get used loosely. A parameter is a number the training loop learns, such as a weight inside the network. A hyperparameter is a number you choose before training starts and the loop never changes: the learning rate, the batch size, how much weight decay to apply, how many epochs to run. Tuning means searching over hyperparameters by training the model repeatedly, which is why the budget is counted in runs rather than in seconds.

One knob matters most#

Learning rate has more effect than every other hyperparameter combined. The learning rate is how far the weights move on each update: too small and the weights barely move, too large and each update overshoots and the run comes apart. Tune it first, roughly, by powers of ten, then refine. Only then look at batch size, schedule and regularisation.

That is the claim everyone repeats. Here it is measured. The setup below is a two-hidden-layer network on a synthetic tabular problem, twenty epochs per run, everything held fixed except the one knob being moved. It runs in about a minute on a laptop and every number in the comments came out of it.

import torch, torch.nn as nn
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

X, y = make_classification(n_samples=12000, n_features=40, n_informative=15,
                           n_redundant=10, class_sep=0.8, flip_y=0.03, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=4000, random_state=0)
Xtr, Xva, ytr, yva = train_test_split(Xtr, ytr, test_size=2000, random_state=0)
scaler = StandardScaler().fit(Xtr)
as_t = lambda a: torch.tensor(scaler.transform(a), dtype=torch.float32)
Xtr, Xva = as_t(Xtr), as_t(Xva)
ytr, yva = torch.tensor(ytr), torch.tensor(yva)

def run(lr=3e-3, wd=0.0, batch=128, width=128, dropout=0.0, epochs=20, seed=0):
    """Train once and return the best validation accuracy seen during the run."""
    torch.manual_seed(seed)
    m = nn.Sequential(nn.Linear(40, width), nn.ReLU(), nn.Dropout(dropout),
                      nn.Linear(width, width), nn.ReLU(), nn.Dropout(dropout),
                      nn.Linear(width, 2))
    opt = torch.optim.AdamW(m.parameters(), lr=lr, weight_decay=wd)
    g = torch.Generator().manual_seed(seed)
    best = 0.0
    for _ in range(epochs):
        m.train()
        perm = torch.randperm(len(Xtr), generator=g)
        for i in range(0, len(Xtr), batch):
            idx = perm[i:i + batch]
            opt.zero_grad()
            nn.functional.cross_entropy(m(Xtr[idx]), ytr[idx]).backward()
            opt.step()
        m.eval()
        with torch.no_grad():
            best = max(best, (m(Xva).argmax(1) == yva).float().mean().item())
    return best

for lr in [1e-5, 1e-4, 1e-3, 3e-3, 1e-2, 3e-2, 1e-1, 1.0]:
    print(lr, round(run(lr=lr), 4))

learning rate   best validation accuracy
  0.00001              0.6625    barely moved off the starting point
  0.0001               0.8860    learning, far too slowly
  0.001                0.9180
  0.003                0.9235
  0.01                 0.9340    best
  0.03                 0.9345    best
  0.1                  0.8990    peaked at epoch 4 then collapsed to 0.5060
  1.0                  0.5080    never learned anything

Two distinct failure modes sit at the ends of that table, and they look nothing alike. At 0.00001 the run is clean, the loss falls smoothly, nothing looks wrong, and the model is simply not finished learning after twenty epochs. That is the one that gets mistaken for a bad idea. At 1.0 the loss is noise from the first few hundred steps. The row at 0.1 is the dangerous one: its best validation accuracy was 0.8990 at epoch four and its final accuracy was 0.5060, so a script that reports only the last epoch records a total failure while a script that reports the best epoch records something respectable. Decide which you are reporting before you read the table.

What the other knobs are worth#

The same harness, with the learning rate pinned at 0.003 and one other knob moved at a time. The last line is the yardstick that makes the rest of the table readable: the same configuration run with five different random seeds.

for wd in [0.0, 1e-4, 1e-2, 1e-1]:      print("wd", wd, round(run(wd=wd), 4))
for bs in [32, 128, 512, 2048]:         print("batch", bs, round(run(batch=bs), 4))
for w in [32, 128, 512]:                print("width", w, round(run(width=w), 4))
for dp in [0.0, 0.1, 0.3, 0.5]:         print("dropout", dp, round(run(dropout=dp), 4))
print("seeds", [round(run(seed=s), 4) for s in range(5)])

knob            values tried                 span of results
learning rate   1e-5 .. 1.0                 0.5080 -> 0.9345    42.65 points
dropout         0.0, 0.1, 0.3, 0.5          0.9235 -> 0.9470     2.35 points
batch size      32, 128, 512, 2048          0.9130 -> 0.9405     2.75 points
width           32, 128, 512                0.9220 -> 0.9305     0.85 points
weight decay    0, 1e-4, 1e-2, 1e-1         0.9235 -> 0.9275     0.40 points

five seeds, identical settings:            0.9235 -> 0.9310     0.75 points

Read that table the way you would read an ablation. Weight decay moved the result by 0.40 points across three orders of magnitude, and the seed alone moved it by 0.75, so on this problem weight decay is not a knob, it is noise with a name. Width is the same story. Dropout and batch size are worth two to three points each, which is real but small, and both are worth roughly a seventeenth of what the learning rate was worth. This is the whole argument for tuning in order, and it is why a sweep that varies six things at once spends most of its compute measuring nothing.

The ordering below generalises better than the exact numbers do. Your problem will have different spans; the shape, one dominant knob and a long tail of small ones, is close to universal for neural networks.

1. learning rate       orders of magnitude, then refine
2. schedule            warmup and decay
3. batch size          interacts with learning rate
4. regularisation      weight decay, dropout
5. architecture size   width and depth
6. everything else     rarely worth the compute

Random beats grid#

This surprises people. With a fixed budget, random search finds better configurations than grid search, because most hyperparameters barely matter. A grid spends its budget evenly across all dimensions, so it tries only a few distinct values of the one that matters. Random search tries a different value of every parameter on every run.

Grid, 9 runs over two parameters:
  3 distinct learning rates, each tried 3 times

Random, 9 runs:
  9 distinct learning rates

If learning rate is what matters, random explored it three
times as thoroughly for the same compute.

That is the argument. The measurement is less dramatic than the argument, and it is worth seeing the honest version. Sixteen runs over learning rate and weight decay, eight epochs each: as a four-by-four grid, and as sixteen points drawn log-uniformly from the same ranges, repeated five times with different draws.

import numpy as np

lrs = [1e-4, 1e-3, 1e-2, 1e-1]
wds = [1e-5, 1e-4, 1e-3, 1e-2]
grid = max(run(lr=lr, wd=wd, epochs=8) for lr in lrs for wd in wds)

rng = np.random.default_rng(0)
for trial in range(5):
    pts = [(10 ** rng.uniform(-4, -1), 10 ** rng.uniform(-5, -2)) for _ in range(16)]
    print(trial, round(max(run(lr=lr, wd=wd, epochs=8) for lr, wd in pts), 4))
print("grid", round(grid, 4))

grid, 16 runs, 4 distinct learning rates      best 0.9310   (at lr = 0.01)
random, 16 runs, 16 distinct learning rates   best 0.9360, 0.9295, 0.9290,
                                                   0.9295, 0.9365
                                              mean 0.9321, won 4 draws of 5

So random search won by about a tenth of a point here, which is inside the seed noise measured earlier. Be honest about that: in two dimensions a sensible grid is fine, because four rungs an order of magnitude apart bracket the optimum well enough. The argument bites in higher dimensions, and it is arithmetic rather than opinion. Five hyperparameters at four values each is 1,024 runs for a full grid, and it still only ever tries four learning rates. Sixteen random draws over the same five dimensions try sixteen learning rates, sixteen batch sizes and sixteen dropout rates. When the budget is smaller than the grid, the grid is not an option and the question does not arise.

Twenty runs: where they should go#

The real question in an interview is not which algorithm, it is how you would spend a fixed budget. Twenty runs on a problem you have not tuned before divide roughly like this.

1. 1.Two runs to confirm the model trains at all with default settings, and that the pipeline is not silently broken. A sweep over a broken model is twenty runs wasted.
2. 2.Five or six runs sweeping the learning rate by powers of ten, then one more to refine between the best two.
3. 3.Six to eight runs over the next two knobs, chosen by what your problem actually is: dropout and augmentation when data is scarce, batch size when throughput matters, weight decay if you have reason to believe it does something here.
4. 4.Three or four runs re-running the top two configurations with different seeds, before declaring a winner.

That last line is the one people cut when the budget is tight, and it is the one that determines whether the result is real. The next section explains why.

When runs are expensive#

Bayesian optimisation builds a model of which regions look promising and samples there. It beats random search when each run costs hours, and the overhead is not worth it when runs are cheap. The rough threshold is whether the time to fit the surrogate model and pick the next point is small compared with a training run; under a few minutes per run it almost never pays.

Early stopping of unpromising trials, as in successive halving, is usually the bigger win: start many configurations, kill the ones falling behind, and give the survivors more budget. Measured on the same problem, over sixteen configurations drawn at random from learning rate, weight decay, batch size, width and dropout. The uniform arm trains the first eight of those sixteen to completion; the halving arm starts all sixteen.

uniform     8 configurations x 16 epochs        128 epoch-units   best 0.9430
halving     16 -> 8 -> 4 -> 2, at 2/2/4/8 epochs   80 epoch-units   best 0.9400

The honest reading is that halving bought the same answer for less, not a better answer. That is what it is for. Its known weakness is configurations that start slowly and finish well, which it kills in the first round; a warmup-heavy schedule or a large model on a small learning rate can both look terrible at epoch two. If your training curves cross late, raise the first rung.

Three splits, not two#

The size of that leak is computable, and it is larger than people expect. Take fifty configurations that are genuinely identical in quality, all of them truly 93.0% accurate, each scored on the same 2,000-example validation set. The score of any one of them has a standard deviation of about 0.57 points from the sampling of the validation set alone. You then report the best of the fifty.

import numpy as np
rng = np.random.default_rng(0)
for k in [10, 20, 50, 200]:
    best = (rng.binomial(2000, 0.930, size=(200_000, k)) / 2000).max(1)
    print(k, round(float(best.mean()), 4))
# 10   0.9387      +0.87 points over the truth
# 20   0.9404      +1.04
# 50   0.9425      +1.25
# 200  0.9452      +1.52

So the winner of a fifty-trial sweep will, on average, look 1.25 points better than it is, purely from having been chosen as the maximum of fifty noisy estimates. This is why the validation score of the winning configuration is not a number you may report, and why the gap between the first and second configuration is usually not a real difference. Two defences, and you want both: hold out a test set that no tuning decision ever touched, and re-run the top two configurations on fresh seeds before choosing between them.

What to skip#

• Do not tune the optimiser choice early. AdamW with its default betas is a fine starting point for almost everything, and swapping optimisers before the learning rate is right compares two badly tuned runs.
• Do not tune on a linear scale. Learning rate, weight decay and any other multiplicative quantity are searched on a log scale; the distance from 0.0001 to 0.001 matters as much as from 0.001 to 0.01.
• Do not tune architecture width and depth before the learning rate, because changing the size changes which learning rate is right and invalidates the comparison.
• Do not carry a hyperparameter across a change of batch size, data size or hardware without re-checking the learning rate.
• Worth memorising: the order of the knobs, the log scale, and that the winning configuration's validation score is inflated by selection.

The learning rate matters more than the rest of the hyperparameters put together, and a single run can find a good one. This section covers that range test, what warmup and cosine decay are for, why changing the batch size means retuning the rate, and then how to spend a fixed budget of runs: a search on log scales with unpromising trials stopped early, in a set order, ending with several seeds on the top two settings. That last step is what stops you shipping a winner that was noise.

Find the learning rate first#

A learning rate range test trains for a few hundred steps while raising the learning rate exponentially from tiny to huge, and records the loss. Loss falls, then explodes. A good starting rate sits a little below where the loss falls fastest, well before it explodes.

import torch, torch.nn as nn

def range_test(model_fn, batches, lo=1e-6, hi=3.0, steps=400):
    """One run over exactly steps batches. Returns (lr, smoothed loss) per step."""
    m = model_fn()
    opt = torch.optim.AdamW(m.parameters(), lr=lo)
    mult, lr, smooth, out = (hi / lo) ** (1 / steps), lo, None, []
    for x, y in batches:
        for pg in opt.param_groups:
            pg["lr"] = lr
        opt.zero_grad()
        loss = nn.functional.cross_entropy(m(x), y)
        loss.backward()
        opt.step()
        v = loss.item()
        smooth = v if smooth is None else 0.9 * smooth + 0.1 * v
        out.append((lr, smooth))
        lr *= mult
    return out

Run on the tabular network from the tuning section, 400 steps from 0.000001 to 3, this is what comes back.

learning rate    smoothed training loss
  0.00001                 0.697     flat: nothing is happening
  0.001                   0.564     falling
  0.0028                  0.362     steepest descent
  0.026                   0.246     minimum
  0.1                     0.598     climbing back out
  1.0                     0.725     diverged

The test found in one run what the sweep found in eight, which is the point of it. Read the advice about it carefully, though. The common rule is to take an order of magnitude below the minimum, which here would give 0.0026, and the sweep says that setting is worth 0.9235 against 0.9345 at the true optimum. That is a point of accuracy given away. The range test brackets the answer within an order of magnitude; it does not hand you the answer. Use it to decide where to centre a short sweep, not to skip the sweep.

Warmup, then decay#

Large models are often unstable in the first steps, when weights and optimiser statistics are poorly set. Warmup raises the learning rate from near zero over the first steps. After that, decaying it, commonly along a cosine curve, lets training settle into a good minimum.

import math

def lr_at(step, total, peak=3e-4, warmup=500):
    if step < warmup:
        return peak * step / warmup
    progress = (step - warmup) / (total - warmup)
    return peak * 0.5 * (1 + math.cos(math.pi * progress))

lr_at(step, total=20000) with peak 3e-4 and 500 warmup steps

  step      0    0.00e+00 the ramp starts at zero
  step    250    1.50e-04 halfway up the warmup ramp
  step    500    3.00e-04 peak
  step   5000    2.62e-04
  step  10000    1.56e-04 halfway through, at 52% of peak
  step  15000    4.61e-05
  step  20000    0.00e+00 the cosine ends at zero

Whether the decay is worth anything is measurable. On the tabular network from the tuning section, cosine decay against a constant rate at the same peak, twenty epochs each.

peak lr   constant: best / final     cosine: best / final
  0.01      0.9340 / 0.9340            0.9370 / 0.9365
  0.03      0.9345 / 0.9230            0.9350 / 0.9320

Read the two columns separately. At a peak of 0.03 the constant-rate run scored 0.9345 at its best epoch and 0.9230 at its last, because a large constant rate keeps bouncing the weights around a minimum it cannot settle into. Decay closed most of that gap. On a small problem you can paper over this with early stopping on the best epoch. On a large training run there is no best epoch to go back to, because you only keep a handful of checkpoints, so the value of the schedule is that the final model is the good one.

Stop when the thing you care about stops improving#

Early stopping means keeping the weights from the epoch with the best validation score rather than the last epoch. What it saves depends on whether the validation curve turns, and the trap is that different validation metrics turn at different times. Here is a deliberately overfitting run: 600 training examples, a 512-wide network, no dropout, 60 epochs.

epoch   train loss   val loss   val accuracy
    1      0.4853      0.5655        0.7035
    3      0.2535      0.4571        0.7820
    5      0.1252      0.4017        0.8235   <- val loss is at its minimum
   10      0.0183      0.4839        0.8400   <- val accuracy is at its maximum
   20      0.0021      0.6276        0.8360
   40      0.0005      0.7440        0.8360
   60      0.0002      0.8113        0.8365

Stopping on validation loss would have stopped at epoch 5 and handed you a model 1.65 accuracy points worse than the one at epoch 10. Both curves are correct: the model is becoming more confident and more often wrong about the cases it gets wrong, which hurts cross-entropy, while its top-1 decisions keep improving slightly. Stop on the metric the product is judged by. If that metric is noisy, use patience, meaning stop after N epochs with no improvement rather than at the first downturn, and pick N from how bumpy your validation curve is rather than from a blog post.

Batch size and learning rate move together#

A larger batch gives a less noisy gradient, which usually allows a larger learning rate. If you change the batch size, retune the learning rate. Comparing two batch sizes at the same learning rate compares two badly tuned runs.

validation accuracy, 20 epochs, learning rate across the top

batch     1e-4    3e-4    1e-3    3e-3    1e-2    3e-2    1e-1
   32    0.914   0.918   0.925   0.941   0.932   0.918   0.506
  128    0.886   0.916   0.918   0.924   0.934   0.934   0.899
  512    0.769   0.896   0.916   0.922   0.928   0.927   0.919
 2048    0.663   0.735   0.882   0.913   0.925   0.926   0.886

Two things in that grid are worth more than the headline. First, 0.1 destroys the batch-32 run, landing it on 0.506, and the same rate is perfectly usable at batch 512. Anyone who fixes the learning rate and then sweeps batch size will conclude that small batches are unstable, which is a statement about their sweep. Second, 0.0001 costs the batch-2048 run 26 points against its own best and the batch-32 run under three points, because at a fixed number of epochs a larger batch takes proportionally fewer steps. The rule of thumb that survives is to move the learning rate in the same direction as the batch size, roughly linearly for moderate changes, and then check.

A search that stops bad runs early#

import optuna

def objective(trial):
    lr = trial.suggest_float("lr", 1e-5, 1e-2, log=True)
    wd = trial.suggest_float("weight_decay", 1e-6, 1e-1, log=True)
    model = build(lr=lr, weight_decay=wd)
    for epoch in range(20):
        score = train_and_validate(model)
        trial.report(score, epoch)
        if trial.should_prune():          # clearly worse than earlier trials at this point
            raise optuna.TrialPruned()
    return score

study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=50)
print(study.best_params)

Search learning rates and weight decay on a log scale; the difference between 0.001 and 0.01 matters as much as between 0.0001 and 0.001. Pruning stops unpromising runs after a few epochs, so the budget goes to the good ones.

The saving from pruning is worth estimating before you set it up. Fifty trials at twenty epochs each is 1,000 epoch-units. Suppose pruning kills 25 of them at epoch two, and 12 of the survivors at epoch five, leaving 13 to run in full. The cost is 25 times 2, plus 12 times 5, plus 13 times 20, which is 370 epoch-units: 63% of the budget came back. The saving grows with the number of trials, and shrinks when good and bad configurations only separate late in training, which is exactly the case where pruning also throws away the winner.

Spend the budget in this order#

1. 1.Get a baseline training cleanly with default settings.
2. 2.Tune the learning rate.
3. 3.Tune regularisation: weight decay, dropout, augmentation strength.
4. 4.Only then touch architecture choices.
5. 5.Rerun the best setting with several seeds before believing it beat the runner-up.

What to skip: warmup on a small model trained with a modest learning rate, where it does nothing measurable and costs you steps; exotic schedules, since cosine with a short warmup is within noise of nearly all of them on nearly all problems; and any schedule tuning at all before the peak learning rate is right, because the schedule is a refinement of the peak and moving both at once tells you which did nothing. Worth memorising: the range test is one run, the cosine ends at zero, and the learning rate moves with the batch size.