Tuning, in order of what matters

There are dozens of settings you could adjust and almost all of them barely matter. This section is about doing it in order: the learning rate dominates everything else, random sampling beats an exhaustive grid for the same budget, and the more expensive each run is the more it is worth being clever about which to run next. The discipline that matters most is never tuning against the set you report on.

Two words before anything else, because they get used loosely. A parameter is a number the training loop learns, such as a weight inside the network. A hyperparameter is a number you choose before training starts and the loop never changes: the learning rate, the batch size, how much weight decay to apply, how many epochs to run. Tuning means searching over hyperparameters by training the model repeatedly, which is why the budget is counted in runs rather than in seconds.

One knob matters most#

Learning rate has more effect than every other hyperparameter combined. The learning rate is how far the weights move on each update: too small and the weights barely move, too large and each update overshoots and the run comes apart. Tune it first, roughly, by powers of ten, then refine. Only then look at batch size, schedule and regularisation.

That is the claim everyone repeats. Here it is measured. The setup below is a two-hidden-layer network on a synthetic tabular problem, twenty epochs per run, everything held fixed except the one knob being moved. It runs in about a minute on a laptop and every number in the comments came out of it.

import torch, torch.nn as nn
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

X, y = make_classification(n_samples=12000, n_features=40, n_informative=15,
                           n_redundant=10, class_sep=0.8, flip_y=0.03, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=4000, random_state=0)
Xtr, Xva, ytr, yva = train_test_split(Xtr, ytr, test_size=2000, random_state=0)
scaler = StandardScaler().fit(Xtr)
as_t = lambda a: torch.tensor(scaler.transform(a), dtype=torch.float32)
Xtr, Xva = as_t(Xtr), as_t(Xva)
ytr, yva = torch.tensor(ytr), torch.tensor(yva)

def run(lr=3e-3, wd=0.0, batch=128, width=128, dropout=0.0, epochs=20, seed=0):
    """Train once and return the best validation accuracy seen during the run."""
    torch.manual_seed(seed)
    m = nn.Sequential(nn.Linear(40, width), nn.ReLU(), nn.Dropout(dropout),
                      nn.Linear(width, width), nn.ReLU(), nn.Dropout(dropout),
                      nn.Linear(width, 2))
    opt = torch.optim.AdamW(m.parameters(), lr=lr, weight_decay=wd)
    g = torch.Generator().manual_seed(seed)
    best = 0.0
    for _ in range(epochs):
        m.train()
        perm = torch.randperm(len(Xtr), generator=g)
        for i in range(0, len(Xtr), batch):
            idx = perm[i:i + batch]
            opt.zero_grad()
            nn.functional.cross_entropy(m(Xtr[idx]), ytr[idx]).backward()
            opt.step()
        m.eval()
        with torch.no_grad():
            best = max(best, (m(Xva).argmax(1) == yva).float().mean().item())
    return best

for lr in [1e-5, 1e-4, 1e-3, 3e-3, 1e-2, 3e-2, 1e-1, 1.0]:
    print(lr, round(run(lr=lr), 4))

learning rate   best validation accuracy
  0.00001              0.6625    barely moved off the starting point
  0.0001               0.8860    learning, far too slowly
  0.001                0.9180
  0.003                0.9235
  0.01                 0.9340    best
  0.03                 0.9345    best
  0.1                  0.8990    peaked at epoch 4 then collapsed to 0.5060
  1.0                  0.5080    never learned anything

Two distinct failure modes sit at the ends of that table, and they look nothing alike. At 0.00001 the run is clean, the loss falls smoothly, nothing looks wrong, and the model is simply not finished learning after twenty epochs. That is the one that gets mistaken for a bad idea. At 1.0 the loss is noise from the first few hundred steps. The row at 0.1 is the dangerous one: its best validation accuracy was 0.8990 at epoch four and its final accuracy was 0.5060, so a script that reports only the last epoch records a total failure while a script that reports the best epoch records something respectable. Decide which you are reporting before you read the table.

What the other knobs are worth#

The same harness, with the learning rate pinned at 0.003 and one other knob moved at a time. The last line is the yardstick that makes the rest of the table readable: the same configuration run with five different random seeds.

for wd in [0.0, 1e-4, 1e-2, 1e-1]:      print("wd", wd, round(run(wd=wd), 4))
for bs in [32, 128, 512, 2048]:         print("batch", bs, round(run(batch=bs), 4))
for w in [32, 128, 512]:                print("width", w, round(run(width=w), 4))
for dp in [0.0, 0.1, 0.3, 0.5]:         print("dropout", dp, round(run(dropout=dp), 4))
print("seeds", [round(run(seed=s), 4) for s in range(5)])

knob            values tried                 span of results
learning rate   1e-5 .. 1.0                 0.5080 -> 0.9345    42.65 points
dropout         0.0, 0.1, 0.3, 0.5          0.9235 -> 0.9470     2.35 points
batch size      32, 128, 512, 2048          0.9130 -> 0.9405     2.75 points
width           32, 128, 512                0.9220 -> 0.9305     0.85 points
weight decay    0, 1e-4, 1e-2, 1e-1         0.9235 -> 0.9275     0.40 points

five seeds, identical settings:            0.9235 -> 0.9310     0.75 points

Read that table the way you would read an ablation. Weight decay moved the result by 0.40 points across three orders of magnitude, and the seed alone moved it by 0.75, so on this problem weight decay is not a knob, it is noise with a name. Width is the same story. Dropout and batch size are worth two to three points each, which is real but small, and both are worth roughly a seventeenth of what the learning rate was worth. This is the whole argument for tuning in order, and it is why a sweep that varies six things at once spends most of its compute measuring nothing.

The ordering below generalises better than the exact numbers do. Your problem will have different spans; the shape, one dominant knob and a long tail of small ones, is close to universal for neural networks.

1. learning rate       orders of magnitude, then refine
2. schedule            warmup and decay
3. batch size          interacts with learning rate
4. regularisation      weight decay, dropout
5. architecture size   width and depth
6. everything else     rarely worth the compute

Random beats grid#

This surprises people. With a fixed budget, random search finds better configurations than grid search, because most hyperparameters barely matter. A grid spends its budget evenly across all dimensions, so it tries only a few distinct values of the one that matters. Random search tries a different value of every parameter on every run.

Grid, 9 runs over two parameters:
  3 distinct learning rates, each tried 3 times

Random, 9 runs:
  9 distinct learning rates

If learning rate is what matters, random explored it three
times as thoroughly for the same compute.

That is the argument. The measurement is less dramatic than the argument, and it is worth seeing the honest version. Sixteen runs over learning rate and weight decay, eight epochs each: as a four-by-four grid, and as sixteen points drawn log-uniformly from the same ranges, repeated five times with different draws.

import numpy as np

lrs = [1e-4, 1e-3, 1e-2, 1e-1]
wds = [1e-5, 1e-4, 1e-3, 1e-2]
grid = max(run(lr=lr, wd=wd, epochs=8) for lr in lrs for wd in wds)

rng = np.random.default_rng(0)
for trial in range(5):
    pts = [(10 ** rng.uniform(-4, -1), 10 ** rng.uniform(-5, -2)) for _ in range(16)]
    print(trial, round(max(run(lr=lr, wd=wd, epochs=8) for lr, wd in pts), 4))
print("grid", round(grid, 4))

grid, 16 runs, 4 distinct learning rates      best 0.9310   (at lr = 0.01)
random, 16 runs, 16 distinct learning rates   best 0.9360, 0.9295, 0.9290,
                                                   0.9295, 0.9365
                                              mean 0.9321, won 4 draws of 5

So random search won by about a tenth of a point here, which is inside the seed noise measured earlier. Be honest about that: in two dimensions a sensible grid is fine, because four rungs an order of magnitude apart bracket the optimum well enough. The argument bites in higher dimensions, and it is arithmetic rather than opinion. Five hyperparameters at four values each is 1,024 runs for a full grid, and it still only ever tries four learning rates. Sixteen random draws over the same five dimensions try sixteen learning rates, sixteen batch sizes and sixteen dropout rates. When the budget is smaller than the grid, the grid is not an option and the question does not arise.

Twenty runs: where they should go#

The real question in an interview is not which algorithm, it is how you would spend a fixed budget. Twenty runs on a problem you have not tuned before divide roughly like this.

1. 1.Two runs to confirm the model trains at all with default settings, and that the pipeline is not silently broken. A sweep over a broken model is twenty runs wasted.
2. 2.Five or six runs sweeping the learning rate by powers of ten, then one more to refine between the best two.
3. 3.Six to eight runs over the next two knobs, chosen by what your problem actually is: dropout and augmentation when data is scarce, batch size when throughput matters, weight decay if you have reason to believe it does something here.
4. 4.Three or four runs re-running the top two configurations with different seeds, before declaring a winner.

That last line is the one people cut when the budget is tight, and it is the one that determines whether the result is real. The next section explains why.

When runs are expensive#

Bayesian optimisation builds a model of which regions look promising and samples there. It beats random search when each run costs hours, and the overhead is not worth it when runs are cheap. The rough threshold is whether the time to fit the surrogate model and pick the next point is small compared with a training run; under a few minutes per run it almost never pays.

Early stopping of unpromising trials, as in successive halving, is usually the bigger win: start many configurations, kill the ones falling behind, and give the survivors more budget. Measured on the same problem, over sixteen configurations drawn at random from learning rate, weight decay, batch size, width and dropout. The uniform arm trains the first eight of those sixteen to completion; the halving arm starts all sixteen.

uniform     8 configurations x 16 epochs        128 epoch-units   best 0.9430
halving     16 -> 8 -> 4 -> 2, at 2/2/4/8 epochs   80 epoch-units   best 0.9400

The honest reading is that halving bought the same answer for less, not a better answer. That is what it is for. Its known weakness is configurations that start slowly and finish well, which it kills in the first round; a warmup-heavy schedule or a large model on a small learning rate can both look terrible at epoch two. If your training curves cross late, raise the first rung.

Three splits, not two#

The size of that leak is computable, and it is larger than people expect. Take fifty configurations that are genuinely identical in quality, all of them truly 93.0% accurate, each scored on the same 2,000-example validation set. The score of any one of them has a standard deviation of about 0.57 points from the sampling of the validation set alone. You then report the best of the fifty.

import numpy as np
rng = np.random.default_rng(0)
for k in [10, 20, 50, 200]:
    best = (rng.binomial(2000, 0.930, size=(200_000, k)) / 2000).max(1)
    print(k, round(float(best.mean()), 4))
# 10   0.9387      +0.87 points over the truth
# 20   0.9404      +1.04
# 50   0.9425      +1.25
# 200  0.9452      +1.52

So the winner of a fifty-trial sweep will, on average, look 1.25 points better than it is, purely from having been chosen as the maximum of fifty noisy estimates. This is why the validation score of the winning configuration is not a number you may report, and why the gap between the first and second configuration is usually not a real difference. Two defences, and you want both: hold out a test set that no tuning decision ever touched, and re-run the top two configurations on fresh seeds before choosing between them.

What to skip#

• Do not tune the optimiser choice early. AdamW with its default betas is a fine starting point for almost everything, and swapping optimisers before the learning rate is right compares two badly tuned runs.
• Do not tune on a linear scale. Learning rate, weight decay and any other multiplicative quantity are searched on a log scale; the distance from 0.0001 to 0.001 matters as much as from 0.001 to 0.01.
• Do not tune architecture width and depth before the learning rate, because changing the size changes which learning rate is right and invalidates the comparison.
• Do not carry a hyperparameter across a change of batch size, data size or hardware without re-checking the learning rate.
• Worth memorising: the order of the knobs, the log scale, and that the winning configuration's validation score is inflated by selection.