Reading training curves
By the end: You can look at loss curves and gradient statistics and say what is going wrong and what to try.
- Healthy curves
- Learning rate too high or too low
- Overfitting
- Dead units and vanishing gradients
- What to log
New to neural networks? A neural network is a stack of simple steps that turn numbers into other numbers. Each layer takes the list of numbers it is given, multiplies and adds them in thousands of small ways, and passes the result on; the values doing that multiplying are called weights, and training means adjusting them. Between the layers sits a small bend — a step that is deliberately not a straight line — and without it the entire stack would collapse into one plain sum no matter how many layers you piled up. Why it matters
A training run emits a stream of numbers, and most of what goes wrong in machine learning shows up there long before it shows up in a final score. This section is about reading that stream: what a healthy loss curve looks like, what a learning rate set too high does to it, how overfitting announces itself, and what gradient statistics tell you when the curve alone is ambiguous. It is the difference between waiting an hour for a run to fail and killing it after two minutes knowing what to change.
What healthy looks like#
Training loss falls quickly at first and then more slowly. Validation loss follows it closely, then flattens a little above it. Both are smooth apart from mini-batch noise.
Every curve below came out of the same twenty-line script, with one number changed each time. It is worth producing them yourself once, because a description of a diverging loss is much less memorable than having watched one.
import torch, torch.nn as nn, torch.nn.functional as F
X = torch.randn(2000, 20); w = torch.randn(20, 3)
y = (X @ w).argmax(1) # a learnable problem, by construction
Xtr, ytr, Xva, yva = X[:1500], y[:1500], X[1500:], y[1500:]
def run(lr, steps=400, n=1500):
torch.manual_seed(0)
model = nn.Sequential(nn.Linear(20, 64), nn.ReLU(), nn.Linear(64, 3))
opt = torch.optim.SGD(model.parameters(), lr=lr)
for s in range(steps):
loss = F.cross_entropy(model(Xtr[:n]), ytr[:n])
opt.zero_grad(); loss.backward()
gnorm = torch.nn.utils.clip_grad_norm_(model.parameters(), 1e9) # measure, do not clip
opt.step()
if s in (0, 50, 100, 200, 399):
with torch.no_grad():
val = F.cross_entropy(model(Xva), yva).item()
dead = (model[0](Xtr[:n]).max(0).values <= 0).float().mean().item()
print(s, round(loss.item(), 4), round(val, 4), round(gnorm.item(), 3), round(dead, 3))run(lr=0.5) HEALTHY
step train val grad norm dead units
0 1.1429 1.0489 0.485 0% starts at log(3) = 1.0986
50 0.1412 0.1646 0.071 0%
100 0.0770 0.1052 0.036 0%
399 0.0204 0.0641 0.011 0% val a little above train, both falling
run(lr=0.001) LEARNING RATE TOO LOW
0 1.1429 1.1485 0.485 0%
200 1.1011 1.1065 0.432 0%
399 1.0672 1.0728 0.397 0% 400 steps to move 0.08. A straight line.
run(lr=50) LEARNING RATE TOO HIGH
0 1.1429 30.9919 0.485 0%
50 318239 2.07e+17 59.157 65.6% one huge step, then wreckage
100 5.5275 2.07e+17 0.683 65.6% train recovers, val never does
399 13.9810 2.07e+17 0.644 65.6% two thirds of the units are deadThe third run is the one worth studying, because its training loss comes back down to single digits and would look survivable in a plot that started at step 100. It is not survivable. One step at lr=50 pushed 65.6 per cent of the hidden units to a permanently negative input, and a ReLU there outputs zero with zero gradient forever. The network is now a third of the size you built, the damage happened in a single step, and nothing in the logs says so unless you measure dead units.
Overfitting looks different again, and cannot be seen at all without a validation split. The same script on 40 training rows instead of 1,500:
run(lr=0.5, steps=2000, n=40) OVERFITTING
step train val
0 1.1979 1.0556
100 0.0065 1.2157 train already essentially zero; val has turned
400 0.0011 1.4405
1000 0.0004 1.5850
1999 0.0002 1.6929 still improving on train, still worsening on valNote that the training loss never misbehaves. It falls smoothly to 0.0002 and would be reported as an excellent run by anyone logging one number. The entire signal is in the second column, and the model worth shipping existed for about nine steps. This is the argument for checkpointing on best validation rather than at the end: the end of training is not where the best model is, and on a small dataset it is nowhere near.
A field guide#
Symptom Likely cause Try
Loss jumps around, sometimes explodes learning rate too high lower it, add warmup, clip gradients
Loss falls painfully slowly in a straight line learning rate too low raise it by 3 to 10x
Loss flat from step 0 bug, frozen parameters overfit one batch, check gradients
Train falls, validation rises after a point overfitting early stop, regularise, more data
Loss plateaus, then suddenly drops normal for some tasks be patient before changing things
Loss becomes NaN overflow, log(0) lower LR, bfloat16, stable loss functionsInside the network#
When the loss curve is ambiguous, the next numbers to look at are inside the model. Three are enough to separate most causes, and all three cost a line each.
- Dead ReLUs: units whose input is always negative output zero and never learn again. Track the share of zero activations per layer; a large learning rate is the usual cause.
- Vanishing gradients: gradient norms in early layers far smaller than in late layers. Residual connections, normalisation and good initialisation are the fixes.
- Exploding gradients: sudden spikes in gradient norm. Clipping caps them.
@torch.no_grad()
def probe(model, x):
"""Dead units per layer, and gradient norm per layer. Run it every few hundred steps."""
h = x
for i, layer in enumerate(model):
h = layer(h)
if isinstance(layer, nn.ReLU):
print("layer", i, "dead fraction", round((h.max(0).values <= 0).float().mean().item(), 3))
for name, p in model.named_parameters():
if p.grad is not None:
print(name, "grad norm", round(p.grad.norm().item(), 6))What you measure What it means What to do
dead fraction rising over training learning rate too high lower it; the dead units do not come back
dead fraction high from step 0 initialisation, or a bad bias check the init scheme and the activation
early-layer grad norm << late-layer vanishing gradients residuals, normalisation, He init
grad norm spikes by 100x on some batches a few pathological examples clip, and go and look at those batches
grad norm exactly 0.0 somewhere detached graph, or frozen check requires_grad and no_grad scopes
grad norm steady, loss flat the loss is wrong, not the modelLog these from the start#
Adding logging after a run goes wrong means running it again to find out why. All of this is cheap except the last, and the last is the one that catches the problems the numbers cannot.
- Training and validation loss.
- The learning rate at each step.
- The total gradient norm, and per-layer norms if something is odd.
- A few sample predictions at intervals, read by a person.
- Wall-clock time per step, which is how you notice the data loader has become the bottleneck.
- The step number of the best validation loss so far, and a checkpoint at it.
Read the predictions, not only the loss
A loss of 0.3 is compatible with a model that is excellent on the common case and catastrophic on a slice that matters, and it is compatible with a model that has learned to exploit a labelling artefact. Twenty predictions printed next to their targets every thousand steps costs nothing and is the only item on this list that can tell you the model is solving the wrong problem.
Where people go wrong
Changing three things when a curve looks wrong. Change one, rerun, compare.
If this comes up in an interview
Your validation loss starts rising while training loss keeps falling. What is happening and what do you do?
The model is overfitting: memorising training examples rather than learning patterns that generalise. Stop at the best validation point, then add regularisation such as weight decay, dropout or augmentation, get more data, or reduce model size.
Then they push further
Could it be anything other than overfitting?
Yes. The validation set might differ from training data, there could be leakage making training easier than it should be, or the learning rate schedule might be misbehaving. Check the validation data and a few predictions before assuming.
Build this: Train a small network and deliberately produce each row of the field guide: too high and too low learning rates, overfitting on a tiny subset, and a NaN. Save the plots with notes; they are your reference for real runs.
Watch it explained
This is one section of Neural networks, 18 hours across 4 sections. Read it in full to get the whole topic in order.