Transfer learning

Training a model from scratch takes data and compute most teams do not have. Transfer learning starts from one somebody else already trained and adapts it, which is how nearly all applied vision and language work is actually done. This section covers why a model trained on one task is useful for another, which layers to freeze and when to let them move, why fine-tuning needs a much smaller learning rate than training did, and how to tell early that your domain is too far from the original for any of it to help.

Your situation                                    Do this
a few hundred labelled examples                   linear probe on frozen features
one to ten thousand                               new head, then unfreeze the last block
tens of thousands                                 whole network, per-layer learning rates
your domain looks like the pretraining data       expect most of the benefit
your domain looks nothing like it                 expect some; measure before committing
you have millions of examples and a large budget  training from scratch becomes arguable
you have not yet run the frozen baseline          run it. It takes twenty seconds.

Why it works#

Early layers learn general features. Edges and textures in vision, syntax and common patterns in language. Those transfer across tasks because they are properties of the medium rather than of the task. Only the later layers are specialised, so only those need replacing.

That is the standard account and it is almost right, but it is worth measuring rather than repeating, because the measurement sharpens it usefully. Take a pretrained ResNet-18, cut it at each stage, and fit a plain logistic regression on whatever comes out. The accuracy at each depth says how much usable structure exists there.

Linear probe on the output of each stage, 2,000 CIFAR-10 images:

stage                dims    pretrained    same network, random weights
conv1 (edges)          64      0.2880              0.2685
layer1                 64      0.4195
layer2                128      0.5320
layer3                256      0.6795
layer4 (objects)      512      0.8045              0.4095

Read the two columns against each other. At conv1 the pretrained network beats random weights by two points, which is nearly nothing: edge and colour detectors are so generic that random filters do almost the same job. At layer4 the gap is forty points. So the sentence to keep is not that early layers transfer and late ones do not; it is that early layers are so general they were barely worth learning, and everything pretraining actually bought you is concentrated in the deeper layers.

That reframing is what makes the recipe below make sense. You freeze the early layers because there is nothing there to gain, not because they are precious. You unfreeze the late layers because that is where the representation is specific enough to need adapting to your task. And a linear probe on the final features is strong because those features already encode objects, which is most of what a classifier needs.

The standard recipe#

1. 1.Take a pretrained model and remove its final layer.
2. 2.Attach a new head sized for your task.
3. 3.Freeze everything except the head and train it. Fast, and needs little data.
4. 4.If you need more, unfreeze the top layers and continue at a much lower learning rate.
5. 5.Rarely, unfreeze everything. Only with substantial data.

Run every step of that recipe on the same 2,000 images and the differences are large enough to settle the argument without any theory.

2,000 CIFAR-10 images, ResNet-18, 3 epochs each, same held-out 2,000:

                                              accuracy   time
1  linear probe on frozen features              0.8040     21s
2  new head only, backbone frozen               0.6475     42s
3  then unfreeze layer4, per-layer rates        0.7220     47s
4  everything at lr 1e-3 from step 0            0.6815     96s
5  from scratch, no pretraining                 0.3210    101s

Three things in that table are worth taking seriously. Transfer is worth 48 points over training from scratch, which on 2,000 images is the difference between a usable model and nothing. Step 4, unfreezing everything at a normal learning rate from the first step, scores four points below the staged version while taking twice as long, which is the trap at the bottom of this section, measured. And the linear probe, which took twenty-one seconds and no gradient descent at all, beat every fine-tuned variant.

The probe winning is not a quirk of this dataset. With a few thousand examples there is not enough signal to improve a representation trained on a million images, so fine-tuning mostly perturbs it. The practical rule is that the probe is the number to beat, and if three epochs of fine-tuning has not beaten it, the honest options are to ship the probe or to go and get more labels, not to fine-tune harder.

How much data you need#

Hundreds of examples     head only, everything frozen
Thousands                head plus the top few layers
Tens of thousands        most of the network, low learning rate
Millions                 consider training from scratch, maybe

When the domain is too far#

Transfer works when your data resembles the pretraining data. Medical scans, satellite imagery or audio spectrograms are far from everyday photographs, and there the benefit shrinks. The check is empirical: try the frozen version first, and if it performs near chance, transfer is not helping.

How to tell whether transfer is working at all#

Near chance is a weak test, because a frozen backbone will usually beat chance on almost anything. The question is not whether the number is above chance but whether it is above what you would have got without the pretraining, and that needs a control. Two of them, and both are cheap.

Control                                  What it tells you
probe on the PRETRAINED features         the number to beat with any fine-tuning
probe on the SAME ARCHITECTURE, random   how much of that number is the architecture
                                         rather than the pretraining

import torch.nn as nn
from torchvision.models import resnet18, ResNet18_Weights
from sklearn.linear_model import LogisticRegression

def probe(weights):
    m = resnet18(weights=weights); m.fc = nn.Identity()
    m = m.to(device).eval()
    Ftr, ytr = extract(m, train_loader)          # forward pass only, no gradients
    Fte, yte = extract(m, test_loader)
    return LogisticRegression(max_iter=3000).fit(Ftr, ytr).score(Fte, yte)

print(probe(ResNet18_Weights.DEFAULT))   # 0.8045
print(probe(None))                        # 0.4095  same architecture, never trained

# gap = 0.395. Transfer is doing almost all of the work here.

pretrained probe   random-weights probe   gap     what it means
     0.80                0.41              0.39    transfer is doing the work; use it
     0.62                0.55              0.07    the architecture is doing the work;
                                                   pretraining is nearly irrelevant here
     0.45                0.44              0.01    your domain is too far. Consider a model
                                                   pretrained on something closer, or
                                                   training from scratch with more data
     0.95                0.93              0.02    the task is easy; almost anything works

The second and fourth rows lead to the same decision for opposite reasons, which is why the gap alone is not enough and you need both numbers. A small gap at a low absolute score means transfer failed and the problem is hard. A small gap at a high absolute score means the problem was easy and you should stop spending money on it.

Where the gap is genuinely small, the fix is usually a closer starting point rather than more effort on this one. There are backbones pretrained on medical images, on satellite imagery, on audio spectrograms and on scientific text, and one of those will beat a general photo model on its own domain by more than any amount of fine-tuning will. Self-supervised pretraining on your own unlabelled data is the option when nothing suitable exists, and unlabelled data is usually the one thing you do have.

Almost nobody trains a vision or text model from scratch. You start from one that has already learned general features and adapt it with the few thousand examples you have. This section does that in code, in the order that works: a linear probe on frozen features first, as a baseline that fine-tuning has to beat and often does not beat by much, then a new head, then unfreezing the later layers at a much lower rate. That order also avoids the standard way of wrecking a pretrained network, which is letting a randomly initialised head push large gradients back through it on the first step.

Start with the cheapest version#

Freeze the pretrained network, run your images through it once to get feature vectors, and train a logistic regression on them. This linear probe takes minutes and is a surprisingly strong baseline. Fine-tuning must beat it to be worth the effort.

The word once is the point. Because nothing in the backbone changes, the features are the same on every epoch, so you compute them a single time and then train on a small matrix of numbers instead of on images. That is why it costs seconds rather than hours, and it is also the fastest way to try twenty different classifiers.

import torch, torch.nn as nn
from torchvision.models import resnet18, ResNet18_Weights
from sklearn.linear_model import LogisticRegression

device = "cuda" if torch.cuda.is_available() else "cpu"

backbone = resnet18(weights=ResNet18_Weights.DEFAULT)
backbone.fc = nn.Identity()                       # keep the features, drop the classifier
backbone = backbone.to(device).eval()

@torch.no_grad()                                  # no graph, no gradients, half the memory
def extract(loader):
    F, Y = [], []
    for x, y in loader:
        F.append(backbone(x.to(device)).cpu()); Y.append(y)
    return torch.cat(F).numpy(), torch.cat(Y).numpy()

Ftr, ytr = extract(train_loader)                  # [n, 512] once, then reused
Fte, yte = extract(test_loader)
print(LogisticRegression(max_iter=3000).fit(Ftr, ytr).score(Fte, yte))
# 0.8040 on 2,000 CIFAR-10 images, in 21 seconds

Replace the head and train#

import torch, torch.nn as nn
from torchvision.models import resnet50, ResNet50_Weights

model = resnet50(weights=ResNet50_Weights.DEFAULT)
for p in model.parameters():
    p.requires_grad = False                         # freeze everything

model.fc = nn.Linear(model.fc.in_features, 5)       # new head for 5 classes, trainable

opt = torch.optim.AdamW(model.fc.parameters(), lr=1e-3)

The order of those lines matters. Freezing happens first and the new head is attached afterwards, because a freshly created Linear has requires_grad True by default and would otherwise be frozen along with everything else. On a ResNet-50 head for 5 classes that leaves a fraction of a per cent of the network trainable.

trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
total = sum(p.numel() for p in model.parameters())
print(trainable, total)      # 5,130 of 11,181,642 on a ResNet-18 with 10 classes = 0.046%

for p in m.parameters():
    p.requires_grad = False

before = m.bn1.running_mean.clone()
m.train(); m(torch.randn(4, 3, 224, 224))
print(torch.allclose(before, m.bn1.running_mean))    # False  -> it changed

m.eval();  before = m.bn1.running_mean.clone()
m(torch.randn(4, 3, 224, 224))
print(torch.allclose(before, m.bn1.running_mean))    # True   -> now it is really frozen

Then unfreeze, gently#

Once the new head has trained, unfreeze the later layers and continue with a much lower learning rate. Early layers detect general things like edges and textures and need little change; later layers are more task-specific. Giving earlier layers a smaller learning rate respects that.

for p in model.layer4.parameters():
    p.requires_grad = True

opt = torch.optim.AdamW([
    {"params": model.layer4.parameters(), "lr": 1e-5},
    {"params": model.fc.parameters(),     "lr": 1e-4},
])

A list of dicts is how PyTorch expresses per-group hyperparameters, and lr is not the only one that can vary; weight_decay works the same way, which matters because decaying the pretrained weights towards zero is a way of erasing exactly what you came for.

Per-layer learning rates across the whole network#

Staged unfreezing needs you to decide how many stages and when to move between them. The alternative is to unfreeze everything at once and let the learning rate encode the same intent continuously: smallest at the input, rising by a fixed multiplier per block, largest at the head. This is discriminative fine-tuning, and on a small dataset it usually matches staged unfreezing in less wall-clock time.

groups = [m.conv1, m.bn1, m.layer1, m.layer2, m.layer3, m.layer4, m.fc]
base, decay = 1e-3, 2.6                # 2.6 is the fast.ai default, and as good as any

params = [
    {"params": g.parameters(), "lr": base / decay ** (len(groups) - 1 - i)}
    for i, g in enumerate(groups)
]
opt = torch.optim.AdamW(params)

stage     learning rate    parameters     how much it is allowed to move
conv1       3.2e-06             9,408        essentially nothing
bn1         8.4e-06               128
layer1      2.2e-05           147,968
layer2      5.7e-05           525,568
layer3      1.5e-04         2,099,712
layer4      3.8e-04         8,393,728        the bulk of the network, and most of the change
fc          1.0e-03             5,130        random at step 0, so it needs the full rate

conv1 moves 310 times more slowly than the head.

The head still has a problem this does not solve: it is random, so its first gradients are large and they travel backwards into layers that are now unfrozen. Two standard fixes, and either is enough. Train the head alone for one epoch first, then apply the per-layer rates. Or use a warmup, so the first few hundred steps happen at a fraction of the rate and the head has stopped being random by the time full-size updates arrive.

Use the right preprocessing#

A pretrained model expects inputs prepared the way they were during its training: size, normalisation, tokenisation. The weights object in torchvision, and the tokenizer that ships with a Hugging Face model, provide exactly that. Using your own preprocessing quietly costs accuracy.

weights = ResNet18_Weights.DEFAULT
transform = weights.transforms()          # never write these numbers yourself
print(transform)

# ImageClassification(
#     crop_size=[224]
#     resize_size=[256]
#     mean=[0.485, 0.456, 0.406]          # ImageNet channel means
#     std=[0.229, 0.224, 0.225]
#     interpolation=InterpolationMode.BILINEAR
# )

Getting this wrong is the most common silent loss of accuracy in transfer learning, because nothing errors: the model receives images with the wrong mean and scale, its first layer responds to values it never saw during pretraining, and every feature downstream is slightly off. You lose a few points and blame the dataset. Use the model's own transform for validation, and build your augmentation around its normalisation rather than instead of it.

Forgetting#

Fine-tuning hard on a small dataset can destroy general abilities the model had, known as catastrophic forgetting. Signs are a large jump in training accuracy with poor validation, or failure on inputs slightly different from your training set. Lower learning rates, fewer unfrozen layers, fewer epochs and methods that update only small added weights, such as LoRA, reduce it.

The mechanism is not mysterious. Gradient descent minimises the loss on the data in front of it and has no term for anything else, so weights encoding abilities your dataset never exercises are free to be overwritten in service of the current objective. Nothing in the loss protects them, and the model gives no indication that it has stopped being able to do something you never asked it about.

How to notice                                  What it looks like
hold out data from BEFORE fine-tuning          accuracy on the original task, measured each epoch
evaluate on inputs slightly off-distribution   a cliff rather than a gentle decline
compare against the frozen probe every epoch   fine-tuning drops below its own starting point
track the distance from the original weights   sum of squared differences, growing fast

Defence                        Cost                      How much it helps
lower learning rate            slower convergence         most of the benefit, for free
freeze more layers             less adaptation possible   a lot, and it is one line
fewer epochs, early stopping   may underfit               a lot; forgetting grows with steps
LoRA or an adapter             a little extra code        the base weights never change at all
mix in original-task data      you need that data         the direct fix, when available
weight decay towards the
  ORIGINAL weights, not zero   a custom optimiser step    effective, and rarely implemented

LoRA deserves the emphasis it gets, because it changes the shape of the problem rather than trading against it. The pretrained weights are frozen and a small pair of low-rank matrices is trained alongside them, so the original model is arithmetically untouched and the adaptation is a separate few megabytes you can remove, swap, or serve several of against one base model. That last property is why it is the default for language models, where keeping one copy of a large model and many small adapters is the difference between one deployment and fifty.