Convolutional networks

An image is a grid where nearby pixels mean something together, and a network that treats it as a flat list of numbers throws that away. A convolution is the fix: slide a small filter across the image so the same pattern is recognised wherever it appears. This section is about why that suits images, how filters and pooling stack into a network, why almost nobody trains one from scratch, and where transformers have and have not displaced them.

What a convolution actually computes#

A convolution is a small grid of numbers, called a kernel or a filter, laid over a patch of the image. Multiply each kernel number by the pixel underneath it, add the results, and that single number is one output pixel. Then slide the kernel one pixel across and do it again. The whole operation is a multiply-and-add repeated at every position, with the same kernel numbers used every time.

Work one through by hand. Here is a 6x6 image with a vertical edge in it — the left three columns are 0, the right three are 9 — and a 3x3 kernel that is 1 in its left column, 0 in the middle and minus 1 on the right.

image                    kernel
0 0 0 9 9 9              1  0 -1
0 0 0 9 9 9              1  0 -1
0 0 0 9 9 9              1  0 -1
0 0 0 9 9 9
0 0 0 9 9 9
0 0 0 9 9 9

Top-left position: the kernel covers rows 0-2, columns 0-2, all zeros.
  (1x0 + 0x0 + -1x0) x 3 rows = 0

Slide one right: the kernel now covers columns 1-3, values 0 0 9.
  each row gives 1x0 + 0x0 + -1x9 = -9, three rows  = -27

Slide again: columns 2-4, values 0 9 9.
  each row gives 1x0 + 0x9 + -1x9 = -9, three rows  = -27

Slide again: columns 3-5, values 9 9 9.
  each row gives 1x9 + 0x9 + -1x9 = 0, three rows   = 0

That kernel is an edge detector, and nobody had to tell it where the edge was. It answers zero in the flat regions and minus twenty-seven at the two positions straddling the boundary. Run the same thing in PyTorch and you get the same four numbers.

import torch
import torch.nn.functional as F

row = [0, 0, 0, 9, 9, 9]
img = torch.tensor([row] * 6, dtype=torch.float32)
kernel = torch.tensor([[1., 0., -1.]] * 3)

out = F.conv2d(img[None, None], kernel[None, None])   # add batch and channel dims
print(tuple(out.shape))
print(out[0, 0].to(int).tolist())

# (1, 1, 4, 4)
# [[0, -27, -27, 0], [0, -27, -27, 0], [0, -27, -27, 0], [0, -27, -27, 0]]

In a real network nobody writes those nine numbers. They are parameters, initialised at random and learned by gradient descent, and a trained network's first layer turns out to hold kernels that look very much like edge and colour detectors anyway, because that is what is useful.

Why this beats a dense layer, in numbers#

The claim that convolution saves parameters is worth doing the arithmetic on, because the ratio is larger than people expect. Take a 32x32 colour image, so 3 x 32 x 32 = 3,072 input numbers, and ask for 64 feature maps at the same resolution: 64 x 32 x 32 = 65,536 outputs.

import torch.nn as nn

conv = nn.Conv2d(3, 64, 3, padding=1)        # 3x3 kernel, 3 channels in, 64 out
dense = nn.Linear(3 * 32 * 32, 64 * 32 * 32)  # same input, same number of outputs

c = sum(p.numel() for p in conv.parameters())
d = sum(p.numel() for p in dense.parameters())
print(c, d, round(d / c))

# 1792 201392128 112384

Check the conv number by hand, because you will be asked to. A 3x3 kernel over 3 input channels is 3 x 3 x 3 = 27 weights per output channel, and there are 64 output channels, so 27 x 64 = 1,728 weights plus 64 biases, which is 1,792. Note what does not appear in that sum: the image size. A 3x3 convolution from 3 channels to 64 has 1,792 parameters on a 32x32 image and 1,792 on a 4000x3000 one. The dense layer's count, by contrast, grows with the square of the pixel count — at 224x224 the same comparison is 9,472 parameters against 120 billion.

Parameter count is only half of it. The other half is that a dense layer has no idea two pixels are neighbours, and no idea that a cat in the corner is the same as a cat in the middle. Put a small bright square in one place, then move it, and a convolution produces the same set of responses in a new position while a dense layer's output changes into something unrelated.

base = torch.zeros(1, 1, 16, 16); base[0, 0, 4:7, 4:7] = 1.0
moved = torch.zeros(1, 1, 16, 16); moved[0, 0, 9:12, 9:12] = 1.0

det = nn.Conv2d(1, 1, 3, padding=1, bias=False)
nn.init.constant_(det.weight, 1.0)
a, b = det(base), det(moved)
print(torch.allclose(a.flatten().sort().values, b.flatten().sort().values))

torch.manual_seed(0)
lin = nn.Linear(256, 8)
with torch.no_grad():
    sim = F.cosine_similarity(lin(base.flatten()[None]), lin(moved.flatten()[None]))
print(round(float(sim), 3))

# True      -- the convolution produces the identical set of responses, relocated
# -0.063    -- the dense layer sees two essentially unrelated inputs

Pooling#

Pooling shrinks the spatial size by summarising each small block with one number. Max pooling over 2x2 blocks keeps the largest value in each block, average pooling keeps the mean. It has no parameters at all.

input 4x4                  2x2 max pool      2x2 average pool
1 3 2 4                     6  4              3.75  1.75
5 6 1 0                     7  9              2.50  6.00
7 2 9 8
0 1 3 4

Top-left block is 1 3 / 5 6: the max is 6, the mean is 15/4 = 3.75.

Pooling does two things. It cuts the number of positions by four, so the layers after it are cheaper. And because each later filter now covers a block of the pooled map, it covers four times the area of the original image with the same 3x3 window — which is how a stack of small filters ends up seeing the whole picture. Modern architectures often downsample with a stride-2 convolution instead, which learns how to summarise rather than always taking the maximum, at the cost of 73,728 parameters for a 3x3 layer from 64 to 128 channels where max pooling costs nothing.

What each depth learns#

Stacking convolutions is what turns edge detectors into object detectors, and the mechanism is the growing receptive field: how much of the original image one output unit can see. One 3x3 layer sees 3x3. Two stacked see 5x5, three see 7x7, and each pooling or stride-2 step doubles the step size so it grows much faster after that. A ResNet-18 on a 224x224 image goes through five spatial reductions, and by the last stage each unit's window covers most of the frame.

ResNet-18 at 224x224, measured by running it:

  after conv1     64 channels  112 x 112      9,408 parameters
  after maxpool   64 channels   56 x  56
  layer1          64 channels   56 x  56    147,968
  layer2         128 channels   28 x  28    525,568
  layer3         256 channels   14 x  14  2,099,712
  layer4         512 channels    7 x   7  8,393,728
  avgpool        512 channels    1 x   1
  fc                                         513,000   (1000 classes)

Total: 11,689,512 parameters.

That shape — halve the resolution, double the channels — is the standard design, and the reason is that the early layers have many positions but few distinct things to say about each one (is there an edge here, at what angle), while the late layers have few positions and a great deal to say about each (is this a wheel, a face, a signature). Probing studies and filter visualisations broadly confirm the progression from oriented edges through textures and object parts to whole objects, though it is a tendency rather than a clean division.

The practical use of this is knowing where to cut when you transfer a model. Early layers hold general edge and texture detectors that are useful on almost any image domain. Late layers hold features specific to the pretraining classes. So you keep the early layers, replace the head, and fine-tune from somewhere in the middle depending on how unlike the pretraining data your images are.

Do not train from scratch#

Almost nobody should. Take a model pretrained on a large image corpus, replace the final layer with one sized for your classes, and fine-tune. It needs a fraction of the data and compute and usually beats anything you would train from nothing.

import torchvision

num_classes = 10                    # however many classes you have
model = torchvision.models.resnet18(weights="DEFAULT")

for p in model.parameters():        # freeze the pretrained layers
    p.requires_grad = False

model.fc = nn.Linear(model.fc.in_features, num_classes)   # new head

# Train the head first, then unfreeze the top layers with a
# much lower learning rate if you need more.

Residual connections are what made depth work#

Before 2015, stacking more convolutional layers stopped helping and then started hurting, and not because of overfitting — the training loss got worse too. The fix was to add each block's input back to its output, so the block only has to learn a correction. You can watch the problem directly: build twenty blocks with and without that addition, push one backward pass through, and look at the gradient reaching the first block.

class Blk(nn.Module):
    def __init__(self, c, residual):
        super().__init__()
        self.residual = residual
        self.f = nn.Sequential(nn.Conv2d(c, c, 3, padding=1), nn.ReLU(),
                               nn.Conv2d(c, c, 3, padding=1))

    def forward(self, x):
        return torch.relu(x + self.f(x)) if self.residual else torch.relu(self.f(x))

for residual in (False, True):
    torch.manual_seed(0)
    net = nn.Sequential(*[Blk(16, residual) for _ in range(20)])
    net(torch.randn(4, 16, 16, 16)).pow(2).mean().backward()
    first = net[0].f[0].weight.grad.norm().item()
    last = net[19].f[0].weight.grad.norm().item()
    print(residual, f"{first:.3e}", f"{last:.3e}")

# False 3.543e-19 2.175e-03
# True  2.919e-01 2.105e+00

The reason is visible in the expression. Differentiating x + f(x) gives 1 + f'(x), so every block passes gradient through unchanged plus whatever its own function contributes. Twenty multiplications of numbers below one is what produced 1e-19; twenty additions of one do not shrink anything. This is why ResNet could go to 152 layers, and the same trick sits inside every transformer block you will ever read.

Have transformers replaced them#

Vision transformers cut an image into patches and run ordinary attention over them. Given enough pretraining data they match or beat convolutional networks, and they hold the top of the research benchmarks. They also carry none of the assumptions a convolution builds in — locality, and a pattern meaning the same thing anywhere — so they have to learn those from data, which is why they need much more of it before they pull ahead.

The honest production answer is that the decision is rarely about accuracy. A ViT-B/16 has 86.6 million parameters against ResNet-50's 25.6 million, and on the machine these words were written on a single 224x224 image takes 70 ms through the transformer and 23 ms through the ResNet. If the model has to run on a phone, in a camera, or on a CPU at thirty frames a second, that ratio decides it before accuracy is discussed. Attention also costs the square of the patch count, so raising the input from 224 to 512 pixels makes the attention matrix twenty-seven times larger, while a convolution's cost grows only with the pixel count.

• Reach for a convolutional backbone when data is limited, latency or power is constrained, the images are large, or the domain is unlike anything a public model was pretrained on.
• Reach for a transformer backbone when a strong pretrained checkpoint exists for something near your domain, when you need to relate distant parts of an image, or when you want text-conditioned behaviour such as searching images by description.
• In either case start from pretrained weights. The architecture argument matters far less than whether you are fine-tuning or training from nothing.

What to skip: the history of specific architectures. Knowing what AlexNet, VGG, Inception and DenseNet each contributed is pub-quiz knowledge unless you are reading papers from that decade. What actually transfers is the arithmetic above, residual connections, and knowing which backbone to pick and why.

Classification tells you what is in an image. Most real vision work needs more than that: where each object is, which pixels belong to it, how many there are. This section covers detection and segmentation, the augmentation that stands in for data you do not have, and the metrics these tasks use, which are not accuracy and take some getting used to. It is also where you learn to choose between a fast one-stage detector and a more accurate two-stage one, which is the practical decision most of the time.

Which task do you actually have#

Start here, because the answer changes the labelling budget by an order of magnitude and that is usually the binding constraint. Read the business question and match it to the cheapest task that answers it.

the question being asked                        the task            label cost

"is there a defect in this image"               classification      one click
"how many pallets are on the lorry"             detection           a box each
"which bin should this parcel go in"            classification      one click
"how far is the pedestrian from the kerb"       detection           a box each
"what fraction of the field is diseased"        semantic segmentation  a mask
"how many leaves, and how big is each"          instance segmentation  a mask each
"read the serial number"                        detection + OCR     a box each

Detection#

Classification says what is in an image. Detection says where each object is, with a box and a label. One-stage detectors predict boxes directly and are fast; two-stage detectors propose regions then classify them and are more accurate. For most applications a one-stage detector is the practical default.

The mechanism worth understanding, because it is where the surprises live, is that every detector predicts far more boxes than there are objects. A one-stage detector emits a prediction at every position of several feature maps — tens of thousands of candidates for an image containing three things — and then a filtering step keeps the best and discards the duplicates. That step is non-maximum suppression: sort by confidence, keep the top box, throw away anything overlapping it by more than a threshold, repeat.

import torch
from torchvision.ops import nms, box_iou

boxes = torch.tensor([[10., 10., 50., 50.],      # three near-duplicate boxes
                      [12., 12., 52., 52.],      # on the same object
                      [14.,  8., 54., 48.],
                      [200., 200., 240., 240.],  # and two on a second object
                      [205., 205., 245., 245.]])
scores = torch.tensor([0.95, 0.90, 0.85, 0.80, 0.75])

print((box_iou(boxes, boxes) * 100).round().to(int).tolist())
for threshold in (0.9, 0.5, 0.3):
    print(threshold, nms(boxes, scores, threshold).tolist())

# [[100, 82, 75, 0, 0], [82, 100, 75, 0, 0], [75, 75, 100, 0, 0],
#  [0, 0, 0, 100, 62], [0, 0, 0, 62, 100]]
# 0.9 [0, 1, 2, 3, 4]
# 0.5 [0, 3]
# 0.3 [0, 3]

The other thing to know before picking an architecture is that one-stage and two-stage is less of a divide than it used to be. The practical ordering is: a modern one-stage detector for anything real-time or on a device; a two-stage detector when small objects or tight boxes matter more than latency; and an open-vocabulary detector, which takes the class names as text, when your classes change often or you have almost no labelled data.

Segmentation#

Assign a label to every pixel. Semantic segmentation labels pixels by class; instance segmentation also separates individual objects of the same class. Architectures typically downsample to understand the scene, then upsample back to full resolution, with skip connections to recover fine detail.

The skip connections are not an optimisation, they are the reason the output has edges. Downsampling from 512 to 16 pixels across is what lets the network know it is looking at a road rather than grey texture, and it also destroys every boundary. Upsampling alone gives you a blurred blob in roughly the right place. Copying the high-resolution feature maps from the encoder across to the matching decoder stage is what puts the boundary back, and a U-shaped architecture with those copies is still the default for medical and satellite imagery two decades on.

Segmentation is scored differently from classification, and using accuracy here is the standard mistake. If 2 per cent of pixels are tumour, a model predicting no tumour anywhere scores 98 per cent pixel accuracy. The metrics that mean something are IoU per class, also called the Jaccard index, averaged over classes; and the Dice coefficient, which is twice the intersection over the sum of the two areas. Dice is more forgiving on small regions, which is why medical work tends to report it and why you should never compare a Dice number with an IoU number.

Augmentation#

Create new training examples by transforming existing ones: flips, crops, rotations, colour changes. It is one of the most effective regularisers in vision and nearly free. The rule is to only apply transformations that preserve the label.

The size of the effect is easy to underestimate, so here it is measured. The task is two shapes of identical area, a 4x4 square and a 2x8 bar. The training set has every object near the centre of the frame, which is what a dataset collected by one person with one camera setup looks like. The test set has them anywhere.

import torch
import torch.nn as nn

def make(n, centred, seed):
    g = torch.Generator().manual_seed(seed)
    X = torch.zeros(n, 1, 32, 32); y = torch.zeros(n, dtype=torch.long)
    for i in range(n):
        cls = int(torch.randint(0, 2, (1,), generator=g)); y[i] = cls
        lo, hi = (12, 16) if centred else (2, 22)
        r = int(torch.randint(lo, hi, (1,), generator=g))
        c = int(torch.randint(lo, hi, (1,), generator=g))
        if cls == 0: X[i, 0, r:r+4, c:c+4] = 1.0      # square, 16 pixels
        else:        X[i, 0, r:r+2, c:c+8] = 1.0      # bar, also 16 pixels
    return X, y

def shift(X, g):                                      # the augmentation
    out = torch.zeros_like(X)
    for i in range(len(X)):
        dy = int(torch.randint(-10, 11, (1,), generator=g))
        dx = int(torch.randint(-10, 11, (1,), generator=g))
        out[i] = torch.roll(X[i], shifts=(dy, dx), dims=(1, 2))
    return out

Xtr, ytr = make(400, True, 0)          # training: always near the centre
Xte, yte = make(400, False, 1)         # test: anywhere in the frame

def train(augment):
    torch.manual_seed(0)
    m = nn.Sequential(nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
                      nn.Conv2d(8, 16, 3, padding=1), nn.ReLU(),
                      nn.Flatten(), nn.Linear(16 * 16 * 16, 2))
    opt = torch.optim.Adam(m.parameters(), lr=1e-3)
    g = torch.Generator().manual_seed(1)
    for _ in range(300):
        xb = shift(Xtr, g) if augment else Xtr
        opt.zero_grad()
        nn.functional.cross_entropy(m(xb), ytr).backward()
        opt.step()
    with torch.no_grad():
        return ((m(Xtr).argmax(1) == ytr).float().mean().item(),
                (m(Xte).argmax(1) == yte).float().mean().item())

for augment in (False, True):
    print(augment, ["%.3f" % v for v in train(augment)])

# False ['1.000', '0.560']     perfect on training, barely above chance elsewhere
# True  ['1.000', '1.000']

Note also what the fix is not. A convolution is translation-equivariant, so the feature maps move correctly with the object; but the flatten-and-linear head at the end reads absolute positions, and that is what memorised the centre. Replacing the head with global average pooling would have helped for the same reason augmentation did. Augmentation is the general tool because it works whatever the architecture got wrong.

Evaluating detection#

Detection uses intersection over union to decide whether a predicted box matches a real one, and mean average precision across classes and thresholds as the summary. As with any metric, check it against the errors that actually matter in the application.

Two habits make that check real rather than rhetorical. Break the metric down by object size — COCO reports small, medium and large separately for exactly this reason, and a system whose overall mAP is 0.6 can be at 0.15 on the small objects that are the point of the project. And decide up front which error is worse, because average precision summarises the whole confidence curve and you will ship a single threshold. A missed defect and a false alarm are not the same cost, and the threshold, not the model, is where you express that.

What to skip. The architectural details of any particular detector family — anchor box design, feature pyramid arithmetic, the exact assignment rules — are things you read when you have a specific problem the defaults do not solve. What earns its place up front is choosing the right task, understanding the candidate-plus-suppression shape, knowing that segmentation is scored by IoU or Dice and never by accuracy, and treating augmentation as a first-class part of the model rather than a finishing touch.

A convolution slides a small window of weights across an image, looking for the same pattern everywhere in it, which is why networks built from them work on pictures. This section covers the practical half of that: what size comes out of a layer, how many parameters it holds, and how much of the image a single output unit can see. Then it builds and trains a small network with the two additions that make deep ones trainable at all, batch normalisation and residual connections.

The arithmetic#

output size = floor((input - kernel + 2 x padding) / stride) + 1

32x32 input, 3x3 kernel, padding 1, stride 1  ->  32x32   (padding 1 keeps the size)
32x32 input, 3x3 kernel, padding 1, stride 2  ->  16x16

parameters = kernel x kernel x in_channels x out_channels + out_channels (bias)
3x3 conv, 64 -> 128 channels:  3 x 3 x 64 x 128 + 128 = 73,856

Three quantities go in and two come out, and it is worth being deliberate about each of them, because shape confusion is where most people stall. The kernel size is how wide the window is. The stride is how far the window jumps between positions, so stride 2 halves the output. The padding is how many rows and columns of zeros are added round the edge before sliding starts, and the reason it exists is that without it every layer shrinks the image a little, which is why padding of half the kernel size — 1 for a 3x3, 2 for a 5x5, 3 for a 7x7 — is the near-universal choice: it keeps the size exactly the same.

Channels are the part people get wrong, because they behave nothing like the spatial dimensions. A convolution does not slide over channels. It reads all of them at once: a 3x3 kernel applied to a 64-channel input is really a 3 x 3 x 64 block of weights, and one such block produces one output channel. The number of output channels is simply how many different blocks you asked for, and it is a free choice — it has no relationship to the input channel count.

import torch
import torch.nn as nn

x = torch.zeros(1, 3, 32, 32)          # batch, channels, height, width
for k, s, p in [(3, 1, 1), (3, 1, 0), (3, 2, 1), (5, 1, 2), (7, 2, 3), (1, 1, 0)]:
    c = nn.Conv2d(3, 64, k, stride=s, padding=p)
    n = sum(q.numel() for q in c.parameters())
    print(f"k={k} s={s} p={p} -> {tuple(c(x).shape)}  params={n}")

# k=3 s=1 p=1 -> (1, 64, 32, 32)  params=1792
# k=3 s=1 p=0 -> (1, 64, 30, 30)  params=1792
# k=3 s=2 p=1 -> (1, 64, 16, 16)  params=1792
# k=5 s=1 p=2 -> (1, 64, 32, 32)  params=4864
# k=7 s=2 p=3 -> (1, 64, 16, 16)  params=9472
# k=1 s=1 p=0 -> (1, 64, 32, 32)  params=256

When the shapes are wrong PyTorch usually tells you, but the message names the tensors rather than the mistake, so it is worth recognising the two you will actually hit.

try:                                    # channels-last data, as image libraries produce
    nn.Conv2d(3, 8, 3)(torch.zeros(1, 32, 32, 3))
except RuntimeError as e:
    print("A:", e)

try:                                    # forgot the spatial size after flattening
    bad = nn.Sequential(nn.Conv2d(3, 8, 3), nn.Flatten(), nn.Linear(8 * 30 * 30, 10))
    bad(torch.zeros(1, 3, 64, 64))
except RuntimeError as e:
    print("B:", e)

# A: Given groups=1, weight of size [8, 3, 3, 3], expected input[1, 32, 32, 3]
#    to have 3 channels, but got 32 channels instead
# B: mat1 and mat2 shapes cannot be multiplied (1x30752 and 7200x10)

Receptive field#

The receptive field is how much of the original image one output unit can see. It matters because a network whose final units see a 30-pixel window cannot recognise anything larger than 30 pixels, however many layers it has. You do not have to trust the formula: put a gradient on one output unit and see which input pixels receive any.

def receptive_field(net, size=41):
    inp = torch.zeros(1, 1, size, size, requires_grad=True)
    out = net(inp)
    mid = out.shape[-1] // 2
    out[0, 0, mid, mid].backward()
    touched = (inp.grad[0, 0].abs() > 0).nonzero()
    return int(touched[:, 0].max() - touched[:, 0].min() + 1)

def ones_conv():
    c = nn.Conv2d(1, 1, 3, padding=1, bias=False)
    nn.init.constant_(c.weight, 1.0)      # so nothing cancels to zero by accident
    return c

for depth in (1, 2, 3, 5, 10):
    print(depth, receptive_field(nn.Sequential(*[ones_conv() for _ in range(depth)])))
print("with a pool:", receptive_field(nn.Sequential(ones_conv(), nn.MaxPool2d(2), ones_conv())))

# 1 3
# 2 5
# 3 7
# 5 11
# 10 21
# with a pool: 7

Two practical consequences. If your objects are large relative to the image, you need downsampling and not merely depth. If your objects are small, aggressive downsampling destroys them before the network can see them, which is why detectors for small objects use higher input resolution or predict from earlier, higher-resolution feature maps.

A small network#

import torch.nn as nn

def block(c_in, c_out):
    return nn.Sequential(
        nn.Conv2d(c_in, c_out, 3, padding=1, bias=False),
        nn.BatchNorm2d(c_out),
        nn.ReLU(),
    )

class SmallCNN(nn.Module):
    def __init__(self, classes=10):
        super().__init__()
        self.net = nn.Sequential(
            block(3, 64), block(64, 64), nn.MaxPool2d(2),       # 32 -> 16
            block(64, 128), block(128, 128), nn.MaxPool2d(2),   # 16 -> 8
            block(128, 256), nn.AdaptiveAvgPool2d(1),           # 8 -> 1
            nn.Flatten(), nn.Linear(256, classes),
        )

    def forward(self, x):          # [B, 3, 32, 32]
        return self.net(x)

Count its parameters by hand before you run it, because this is the arithmetic an interviewer asks for out loud. Each conv is 3 x 3 x in x out with no bias, because the batch norm that follows has its own shift; each batch norm is two numbers per channel.

conv   3 -> 64     3 x 3 x   3 x  64 =   1,728   + 128 batchnorm
conv  64 -> 64     3 x 3 x  64 x  64 =  36,864   + 128
conv  64 -> 128    3 x 3 x  64 x 128 =  73,728   + 256
conv 128 -> 128    3 x 3 x 128 x 128 = 147,456   + 256
conv 128 -> 256    3 x 3 x 128 x 256 = 294,912   + 512
linear 256 -> 10                          2,560   +  10

                                        558,538 total

sum(p.numel() for p in SmallCNN().parameters())  ->  558538

Batch norm#

Batch normalisation rescales each channel's activations to a steady mean and spread, which makes deeper networks train faster and much less sensitively to the learning rate. During training it uses the mean and variance of the current batch; during evaluation it uses a running average accumulated over training. That difference is not a detail, and model.eval() matters here more than anywhere else in PyTorch, because in training mode a prediction for one image depends on which other images happened to be in the batch with it.

torch.manual_seed(0)
net = nn.Sequential(nn.Conv2d(3, 8, 3, padding=1), nn.BatchNorm2d(8), nn.ReLU(),
                    nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(8, 4))
x, y = torch.randn(64, 3, 16, 16), torch.randint(0, 4, (64,))
opt = torch.optim.SGD(net.parameters(), lr=0.1)
for _ in range(50):                          # enough for the running stats to settle
    opt.zero_grad()
    nn.functional.cross_entropy(net(x), y).backward()
    opt.step()

sample = torch.randn(1, 3, 16, 16)           # the image we care about
crowd_a = torch.randn(7, 3, 16, 16)          # seven ordinary neighbours
crowd_b = torch.randn(7, 3, 16, 16) * 4      # seven with a very different spread

for mode in ("train", "eval"):
    getattr(net, mode)()
    with torch.no_grad():
        a = net(torch.cat([sample, crowd_a]))[0]
        b = net(torch.cat([sample, crowd_b]))[0]
    diff = round(float((a - b).abs().max()), 4)
    print(mode, "max difference for the same image:", diff)

# train max difference for the same image: 0.1517
# eval  max difference for the same image: 0.0

Residual connections#

class Residual(nn.Module):
    def __init__(self, c):
        super().__init__()
        self.f = nn.Sequential(block(c, c), nn.Conv2d(c, c, 3, padding=1, bias=False), nn.BatchNorm2d(c))

    def forward(self, x):
        return nn.functional.relu(x + self.f(x))    # add the input back

Adding the input back means each block only has to learn a correction, and gradients have a direct path to early layers. This is what made networks with a hundred or more layers trainable, and the same idea runs through every transformer. There is one detail the code above hides: the addition requires the two tensors to have the same shape, so whenever a block changes the channel count or the stride, the shortcut needs its own 1x1 convolution to match. That is exactly what the downsample branch is in torchvision's ResNet implementation, and it is the first thing to look at when a residual network throws a shape error.

What to skip. You do not need to memorise the list of normalisation variants, or the difference between the original and pre-activation residual orderings, or the schedule that any particular paper used. What you do need, cold, is the output-size formula, the parameter formula, the fact that batch norm behaves differently in the two modes, and why an addition keeps gradients alive.

The architecture that took over language turned out to work on images too, once you cut an image into patches and treat them as a sequence. This section covers how that works, why it needed far more data than a convolutional network before it paid off, and how training on image and caption pairs together produces a model that can classify things nobody labelled for it. It ends on the practical question: which backbone to reach for on a given problem.

Patches as tokens#

A vision transformer cuts an image into fixed-size patches, such as 16x16 pixels, flattens each into a vector, and treats the sequence of patches like a sequence of words. A 224x224 image becomes 196 tokens. From there it is an ordinary transformer.

The patch size is the whole design decision, because it sets the sequence length, and attention costs the square of that. Halving the patch size quadruples the token count and multiplies the attention cost by sixteen. The same happens when you raise the input resolution.

16x16 patches:

  224 px ->  196 patches, attention matrix    38,416 entries      1x
  384 px ->  576 patches, attention matrix   331,776 entries    8.6x
  512 px -> 1024 patches, attention matrix 1,048,576 entries   27.3x
 1024 px -> 4096 patches, attention matrix 16,777,216 entries  436.7x

The data question#

A CNN has built-in assumptions that suit images: nearby pixels matter together, and a pattern means the same thing anywhere. A transformer has none and must learn them from data. With modest data CNNs tend to win; with very large pretraining, or strong augmentation and distillation, transformers match or beat them.

Those assumptions are called an inductive bias, and the trade is a general one: a bias that is true of your data saves you the data needed to discover it, and a bias that is false of your data costs you accuracy you cannot recover. Translation equivariance is true of natural photographs and false of, say, a fixed-camera document scan where position carries meaning, which is one of the rare cases where a transformer's lack of it is an advantage rather than a tax.

The size difference is worth having in your head. A ViT-B/16 holds 86.6 million parameters against ResNet-50's 25.6 million, and one 224x224 image took 70 ms through the transformer and 23 ms through the ResNet on the CPU these words were written on. Those exact milliseconds are a property of this machine and will not match yours; the ratio is the transferable part, and it survives onto a GPU.

Learning from captions#

CLIP-style models train an image encoder and a text encoder together on huge numbers of image and caption pairs. The objective pulls matching pairs close in a shared embedding space and pushes mismatched pairs apart. The result is an image embedding you can compare with any text.

The objective is small enough to write out, and writing it out is the fastest way to see what it does. Take a batch of image and caption pairs. Normalise both sets of vectors, take every image against every caption, and the correct answer for row i is column i — so it is an ordinary cross-entropy over the batch, computed once in each direction.

import torch
import torch.nn.functional as F

def clip_loss(img, txt, temperature=0.07):
    img = F.normalize(img, dim=1)
    txt = F.normalize(txt, dim=1)
    logits = img @ txt.T / temperature       # batch x batch similarity matrix
    target = torch.arange(len(img))          # the diagonal is the true pairing
    return (F.cross_entropy(logits, target) + F.cross_entropy(logits.T, target)) / 2

torch.manual_seed(0)
img = torch.randn(8, 32)
aligned = img + 0.1 * torch.randn(8, 32)     # captions that match their image
shuffled = aligned[torch.randperm(8)]        # right captions, wrong images
noise = torch.randn(8, 32)                   # captions with nothing to do with it

print(round(float(clip_loss(img, aligned)), 4))
print(round(float(clip_loss(img, shuffled)), 4))
print(round(float(clip_loss(img, noise)), 4))
print(round(float(torch.log(torch.tensor(8.0))), 4), "= chance for a batch of 8")

# 0.0      matched pairs, perfectly separated from the seven negatives
# 9.6629   shuffled: worse than chance, because it is confidently wrong
# 2.7001   unrelated
# 2.0794   = chance

The temperature is the other thing to understand, because it appears in every contrastive loss and is usually learned rather than fixed. Dividing the similarities by a small number before the softmax sharpens the distribution, so the loss cares intensely about the single closest negative rather than spreading its attention.

softmax over four similarity scores, same scores, three temperatures:

  temperature 1.00    0.238  0.302  0.207  0.254     nearly uniform, weak signal
  temperature 0.07    0.029  0.893  0.004  0.073     the usual setting
  temperature 0.01    0.000  1.000  0.000  0.000     saturated, gradients vanish

Classification without training#

Because image and text land in one space, you can classify into categories nobody trained on: write each class as a sentence, embed the sentences, and pick the nearest.

# encode_image and encode_text stand in for whichever CLIP implementation you
# installed; open_clip and transformers both expose methods with these names.
labels = ["cat", "dog", "invoice", "receipt"]
texts = ["a photo of a " + label for label in labels]

image_emb = encode_image(img)          # [1, d], normalised
text_emb = encode_text(texts)          # [4, d], normalised
probs = (100 * image_emb @ text_emb.T).softmax(dim=-1)
print(labels[probs.argmax()])

Zero-shot classification is a strong baseline for new categories. When you have labelled data, a linear probe on the image embeddings usually beats it cheaply: freeze the encoder, embed your training set once, and fit a logistic regression on the embeddings. It trains in seconds on a laptop, needs no gradients through the backbone, and gives you a number to beat before anyone spends a week on fine-tuning. It also fails in the obvious place — if your classes are internal product codes or defect types that no caption ever described, zero-shot has nothing to work with and the probe is the only option of the two that works at all.

Which backbone#

• Small data, edge devices, tight latency: a modern CNN.
• A pretrained transformer or CLIP-style encoder is available for your domain: start from it.
• Need to search images by text or classify open-ended categories: a CLIP-style model.
• Large images, or objects that occupy a few pixels: a CNN, or a transformer with windowed attention. Full attention over patches at high resolution is the case the arithmetic above rules out.
• You genuinely do not know: run zero-shot CLIP and a fine-tuned ResNet on the same held-out set this week. Two numbers settle it faster than any argument about inductive bias.

What to skip. The taxonomy of vision transformer variants is a research literature, not an engineering skill, and the names turn over every eighteen months. Learn patches-as-tokens, why the data requirement is higher, the quadratic cost in patch count, and the contrastive objective above. Everything else can be looked up on the day you need a specific checkpoint.

Classification tells you what is in an image. Detection and segmentation tell you where, as boxes or as pixel masks. This section is about doing that with a pretrained model rather than from scratch: the annotation formats you will be handed, how the overlap between a predicted box and a true one is scored, what mean average precision is actually averaging, and how to fine-tune a detector on your own data. It also costs out the labelling, which is usually the real constraint.

The annotation formats you will be handed#

Three formats account for nearly everything, and they disagree about what a box is, which is the source of most wasted afternoons in this work.

COCO     JSON. Boxes are [x, y, width, height] in pixels, top-left origin.
         Classes are integer ids with a separate categories list.

Pascal VOC  One XML file per image. Boxes are xmin, ymin, xmax, ymax in pixels.

YOLO     One text file per image, one line per object:
           class_id  x_centre  y_centre  width  height
         All four numbers are fractions of the image size, not pixels.

torchvision detection models want xmin, ymin, xmax, ymax in pixels.

Intersection over union#

def iou(a, b):                        # boxes as (x1, y1, x2, y2)
    ix1, iy1 = max(a[0], b[0]), max(a[1], b[1])
    ix2, iy2 = min(a[2], b[2]), min(a[3], b[3])
    inter = max(0, ix2 - ix1) * max(0, iy2 - iy1)
    area = lambda r: (r[2] - r[0]) * (r[3] - r[1])
    return inter / (area(a) + area(b) - inter)

truth = (10, 10, 50, 50)              # a 40 x 40 box
for pred in [(10, 10, 50, 50), (12, 12, 52, 52), (20, 20, 60, 60),
             (30, 30, 70, 70), (10, 10, 90, 90)]:
    print(pred, round(iou(truth, pred), 3))

# (10, 10, 50, 50) 1.0     exact
# (12, 12, 52, 52) 0.822   off by 2 pixels in each direction
# (20, 20, 60, 60) 0.391   off by 10, a quarter of the box
# (30, 30, 70, 70) 0.143   off by 30
# (10, 10, 90, 90) 0.25    correct corner, four times the area

A predicted box counts as correct if its IoU with a true box of the same class exceeds a threshold, often 0.5. Note the last row: a box with the right corner and four times the area scores 0.25, so a detector that consistently predicts boxes too large is penalised as severely as one that misses objects entirely.

Mean average precision, worked#

Sort all predictions for a class by confidence, walk down the list, and trace precision against recall. The area under that curve is the average precision for the class. Mean average precision averages it over classes. The COCO convention also averages over IoU thresholds from 0.5 to 0.95, which rewards tight boxes. That is the definition; here it is on three true objects and five predictions.

three objects, five predictions, matched at IoU 0.5, walking down by confidence:

  conf   best IoU   verdict   running precision   running recall
  0.95     0.822      TP           1/1 = 1.000       1/3 = 0.333
  0.90     0.620      TP           2/2 = 1.000       2/3 = 0.667
  0.80     0.000      FP           2/3 = 0.667       2/3 = 0.667
  0.70     0.747      TP           3/4 = 0.750       3/3 = 1.000
  0.60     0.681      FP           3/5 = 0.600       3/3 = 1.000

  AP at IoU 0.50 = 0.917
  AP at IoU 0.75 = 0.333
  AP at IoU 0.90 = 0.000
  AP averaged over 0.50 to 0.95 in steps of 0.05 (COCO mAP) = 0.442

That collapse from 0.917 to 0.333 between IoU 0.5 and 0.75 is the honest state of most detectors: they find the objects and their boxes are loose. Whether that matters is a question about your application, not about the model. If the box drives a robot arm, tightness is the product. If it crops a thumbnail, 0.5 is plenty and reporting COCO mAP will make a perfectly adequate system look broken.

Fine-tune, do not train from scratch#

Start from a detector pretrained on a large dataset such as COCO and fine-tune it on your images. A few hundred well-labelled images per class often go a long way. torchvision ships detection and segmentation models with pretrained weights; several popular libraries make the whole loop a few lines. The change you make is the same one as for classification — replace the head with one sized for your classes — it is just buried a little deeper.

import torch
import torchvision
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor

num_classes = 4                      # your classes PLUS one for background
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(weights="DEFAULT")
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

# In training mode it takes targets and returns a dict of losses, not predictions.
model.train()
images = [torch.rand(3, 240, 320), torch.rand(3, 200, 300)]   # any sizes, unbatched
targets = [{"boxes": torch.tensor([[10., 10., 90., 90.]]), "labels": torch.tensor([1])},
           {"boxes": torch.tensor([[20., 30., 80., 120.]]), "labels": torch.tensor([2])}]
losses = model(images, targets)
print(sorted(losses))
loss = sum(losses.values())          # you add them up yourself

model.eval()
with torch.no_grad():
    out = model([torch.rand(3, 240, 320)])
print(sorted(out[0]), tuple(out[0]["boxes"].shape))

# ['loss_box_reg', 'loss_classifier', 'loss_objectness', 'loss_rpn_box_reg']
# ['boxes', 'labels', 'scores'] (100, 4)

Labels cost the most#

Cost this out before anything else, because it usually decides the project. Drawing a box takes on the order of ten seconds once a labeller is warmed up; a careful polygon mask takes a minute or more. Suppose you want 500 images per class across 4 classes, at an average of 3 objects per image.

2,000 images x 3 objects = 6,000 annotations

  boxes at 10 s each      16.7 hours
  masks at 70 s each     116.7 hours

Add a review pass at half the original rate, and a second labeller on a
10% sample to measure agreement:

  boxes     16.7 + 8.3 + 1.7  =  26.7 hours
  masks    116.7 + 58.3 + 11.7 = 186.7 hours

• Boxes take seconds each; pixel masks take much longer. Choose the cheapest label that answers the business question.
• Write labelling guidelines with examples of edge cases, such as occluded or partly visible objects, and measure agreement.
• Use the model to pre-label and have people correct, once it is decent. Correcting a box is roughly three times faster than drawing one, so this pays for itself well before the model is good.
• Label your validation set first and label it more carefully than the training set. Noise in training data costs you a little accuracy; noise in the validation set means you cannot tell which of two models is better, which costs you every decision afterwards.