Starting from someone else's training run

Training a model from scratch takes data and compute most teams do not have. Transfer learning starts from one somebody else already trained and adapts it, which is how nearly all applied vision and language work is actually done. This section covers why a model trained on one task is useful for another, which layers to freeze and when to let them move, why fine-tuning needs a much smaller learning rate than training did, and how to tell early that your domain is too far from the original for any of it to help.

Your situation                                    Do this
a few hundred labelled examples                   linear probe on frozen features
one to ten thousand                               new head, then unfreeze the last block
tens of thousands                                 whole network, per-layer learning rates
your domain looks like the pretraining data       expect most of the benefit
your domain looks nothing like it                 expect some; measure before committing
you have millions of examples and a large budget  training from scratch becomes arguable
you have not yet run the frozen baseline          run it. It takes twenty seconds.

Why it works#

Early layers learn general features. Edges and textures in vision, syntax and common patterns in language. Those transfer across tasks because they are properties of the medium rather than of the task. Only the later layers are specialised, so only those need replacing.

That is the standard account and it is almost right, but it is worth measuring rather than repeating, because the measurement sharpens it usefully. Take a pretrained ResNet-18, cut it at each stage, and fit a plain logistic regression on whatever comes out. The accuracy at each depth says how much usable structure exists there.

Linear probe on the output of each stage, 2,000 CIFAR-10 images:

stage                dims    pretrained    same network, random weights
conv1 (edges)          64      0.2880              0.2685
layer1                 64      0.4195
layer2                128      0.5320
layer3                256      0.6795
layer4 (objects)      512      0.8045              0.4095

Read the two columns against each other. At conv1 the pretrained network beats random weights by two points, which is nearly nothing: edge and colour detectors are so generic that random filters do almost the same job. At layer4 the gap is forty points. So the sentence to keep is not that early layers transfer and late ones do not; it is that early layers are so general they were barely worth learning, and everything pretraining actually bought you is concentrated in the deeper layers.

That reframing is what makes the recipe below make sense. You freeze the early layers because there is nothing there to gain, not because they are precious. You unfreeze the late layers because that is where the representation is specific enough to need adapting to your task. And a linear probe on the final features is strong because those features already encode objects, which is most of what a classifier needs.

The standard recipe#

1. 1.Take a pretrained model and remove its final layer.
2. 2.Attach a new head sized for your task.
3. 3.Freeze everything except the head and train it. Fast, and needs little data.
4. 4.If you need more, unfreeze the top layers and continue at a much lower learning rate.
5. 5.Rarely, unfreeze everything. Only with substantial data.

Run every step of that recipe on the same 2,000 images and the differences are large enough to settle the argument without any theory.

2,000 CIFAR-10 images, ResNet-18, 3 epochs each, same held-out 2,000:

                                              accuracy   time
1  linear probe on frozen features              0.8040     21s
2  new head only, backbone frozen               0.6475     42s
3  then unfreeze layer4, per-layer rates        0.7220     47s
4  everything at lr 1e-3 from step 0            0.6815     96s
5  from scratch, no pretraining                 0.3210    101s

Three things in that table are worth taking seriously. Transfer is worth 48 points over training from scratch, which on 2,000 images is the difference between a usable model and nothing. Step 4, unfreezing everything at a normal learning rate from the first step, scores four points below the staged version while taking twice as long, which is the trap at the bottom of this section, measured. And the linear probe, which took twenty-one seconds and no gradient descent at all, beat every fine-tuned variant.

The probe winning is not a quirk of this dataset. With a few thousand examples there is not enough signal to improve a representation trained on a million images, so fine-tuning mostly perturbs it. The practical rule is that the probe is the number to beat, and if three epochs of fine-tuning has not beaten it, the honest options are to ship the probe or to go and get more labels, not to fine-tune harder.

How much data you need#

Hundreds of examples     head only, everything frozen
Thousands                head plus the top few layers
Tens of thousands        most of the network, low learning rate
Millions                 consider training from scratch, maybe

When the domain is too far#

Transfer works when your data resembles the pretraining data. Medical scans, satellite imagery or audio spectrograms are far from everyday photographs, and there the benefit shrinks. The check is empirical: try the frozen version first, and if it performs near chance, transfer is not helping.

How to tell whether transfer is working at all#

Near chance is a weak test, because a frozen backbone will usually beat chance on almost anything. The question is not whether the number is above chance but whether it is above what you would have got without the pretraining, and that needs a control. Two of them, and both are cheap.

Control                                  What it tells you
probe on the PRETRAINED features         the number to beat with any fine-tuning
probe on the SAME ARCHITECTURE, random   how much of that number is the architecture
                                         rather than the pretraining

import torch.nn as nn
from torchvision.models import resnet18, ResNet18_Weights
from sklearn.linear_model import LogisticRegression

def probe(weights):
    m = resnet18(weights=weights); m.fc = nn.Identity()
    m = m.to(device).eval()
    Ftr, ytr = extract(m, train_loader)          # forward pass only, no gradients
    Fte, yte = extract(m, test_loader)
    return LogisticRegression(max_iter=3000).fit(Ftr, ytr).score(Fte, yte)

print(probe(ResNet18_Weights.DEFAULT))   # 0.8045
print(probe(None))                        # 0.4095  same architecture, never trained

# gap = 0.395. Transfer is doing almost all of the work here.

pretrained probe   random-weights probe   gap     what it means
     0.80                0.41              0.39    transfer is doing the work; use it
     0.62                0.55              0.07    the architecture is doing the work;
                                                   pretraining is nearly irrelevant here
     0.45                0.44              0.01    your domain is too far. Consider a model
                                                   pretrained on something closer, or
                                                   training from scratch with more data
     0.95                0.93              0.02    the task is easy; almost anything works

The second and fourth rows lead to the same decision for opposite reasons, which is why the gap alone is not enough and you need both numbers. A small gap at a low absolute score means transfer failed and the problem is hard. A small gap at a high absolute score means the problem was easy and you should stop spending money on it.

Where the gap is genuinely small, the fix is usually a closer starting point rather than more effort on this one. There are backbones pretrained on medical images, on satellite imagery, on audio spectrograms and on scientific text, and one of those will beat a general photo model on its own domain by more than any amount of fine-tuning will. Self-supervised pretraining on your own unlabelled data is the option when nothing suitable exists, and unlabelled data is usually the one thing you do have.