Convolutional networks

An image is a grid where nearby pixels mean something together, and a network that treats it as a flat list of numbers throws that away. A convolution is the fix: slide a small filter across the image so the same pattern is recognised wherever it appears. This section is about why that suits images, how filters and pooling stack into a network, why almost nobody trains one from scratch, and where transformers have and have not displaced them.

What a convolution actually computes#

A convolution is a small grid of numbers, called a kernel or a filter, laid over a patch of the image. Multiply each kernel number by the pixel underneath it, add the results, and that single number is one output pixel. Then slide the kernel one pixel across and do it again. The whole operation is a multiply-and-add repeated at every position, with the same kernel numbers used every time.

Work one through by hand. Here is a 6x6 image with a vertical edge in it — the left three columns are 0, the right three are 9 — and a 3x3 kernel that is 1 in its left column, 0 in the middle and minus 1 on the right.

image                    kernel
0 0 0 9 9 9              1  0 -1
0 0 0 9 9 9              1  0 -1
0 0 0 9 9 9              1  0 -1
0 0 0 9 9 9
0 0 0 9 9 9
0 0 0 9 9 9

Top-left position: the kernel covers rows 0-2, columns 0-2, all zeros.
  (1x0 + 0x0 + -1x0) x 3 rows = 0

Slide one right: the kernel now covers columns 1-3, values 0 0 9.
  each row gives 1x0 + 0x0 + -1x9 = -9, three rows  = -27

Slide again: columns 2-4, values 0 9 9.
  each row gives 1x0 + 0x9 + -1x9 = -9, three rows  = -27

Slide again: columns 3-5, values 9 9 9.
  each row gives 1x9 + 0x9 + -1x9 = 0, three rows   = 0

That kernel is an edge detector, and nobody had to tell it where the edge was. It answers zero in the flat regions and minus twenty-seven at the two positions straddling the boundary. Run the same thing in PyTorch and you get the same four numbers.

import torch
import torch.nn.functional as F

row = [0, 0, 0, 9, 9, 9]
img = torch.tensor([row] * 6, dtype=torch.float32)
kernel = torch.tensor([[1., 0., -1.]] * 3)

out = F.conv2d(img[None, None], kernel[None, None])   # add batch and channel dims
print(tuple(out.shape))
print(out[0, 0].to(int).tolist())

# (1, 1, 4, 4)
# [[0, -27, -27, 0], [0, -27, -27, 0], [0, -27, -27, 0], [0, -27, -27, 0]]

In a real network nobody writes those nine numbers. They are parameters, initialised at random and learned by gradient descent, and a trained network's first layer turns out to hold kernels that look very much like edge and colour detectors anyway, because that is what is useful.

Why this beats a dense layer, in numbers#

The claim that convolution saves parameters is worth doing the arithmetic on, because the ratio is larger than people expect. Take a 32x32 colour image, so 3 x 32 x 32 = 3,072 input numbers, and ask for 64 feature maps at the same resolution: 64 x 32 x 32 = 65,536 outputs.

import torch.nn as nn

conv = nn.Conv2d(3, 64, 3, padding=1)        # 3x3 kernel, 3 channels in, 64 out
dense = nn.Linear(3 * 32 * 32, 64 * 32 * 32)  # same input, same number of outputs

c = sum(p.numel() for p in conv.parameters())
d = sum(p.numel() for p in dense.parameters())
print(c, d, round(d / c))

# 1792 201392128 112384

Check the conv number by hand, because you will be asked to. A 3x3 kernel over 3 input channels is 3 x 3 x 3 = 27 weights per output channel, and there are 64 output channels, so 27 x 64 = 1,728 weights plus 64 biases, which is 1,792. Note what does not appear in that sum: the image size. A 3x3 convolution from 3 channels to 64 has 1,792 parameters on a 32x32 image and 1,792 on a 4000x3000 one. The dense layer's count, by contrast, grows with the square of the pixel count — at 224x224 the same comparison is 9,472 parameters against 120 billion.

Parameter count is only half of it. The other half is that a dense layer has no idea two pixels are neighbours, and no idea that a cat in the corner is the same as a cat in the middle. Put a small bright square in one place, then move it, and a convolution produces the same set of responses in a new position while a dense layer's output changes into something unrelated.

base = torch.zeros(1, 1, 16, 16); base[0, 0, 4:7, 4:7] = 1.0
moved = torch.zeros(1, 1, 16, 16); moved[0, 0, 9:12, 9:12] = 1.0

det = nn.Conv2d(1, 1, 3, padding=1, bias=False)
nn.init.constant_(det.weight, 1.0)
a, b = det(base), det(moved)
print(torch.allclose(a.flatten().sort().values, b.flatten().sort().values))

torch.manual_seed(0)
lin = nn.Linear(256, 8)
with torch.no_grad():
    sim = F.cosine_similarity(lin(base.flatten()[None]), lin(moved.flatten()[None]))
print(round(float(sim), 3))

# True      -- the convolution produces the identical set of responses, relocated
# -0.063    -- the dense layer sees two essentially unrelated inputs

Pooling#

Pooling shrinks the spatial size by summarising each small block with one number. Max pooling over 2x2 blocks keeps the largest value in each block, average pooling keeps the mean. It has no parameters at all.

input 4x4                  2x2 max pool      2x2 average pool
1 3 2 4                     6  4              3.75  1.75
5 6 1 0                     7  9              2.50  6.00
7 2 9 8
0 1 3 4

Top-left block is 1 3 / 5 6: the max is 6, the mean is 15/4 = 3.75.

Pooling does two things. It cuts the number of positions by four, so the layers after it are cheaper. And because each later filter now covers a block of the pooled map, it covers four times the area of the original image with the same 3x3 window — which is how a stack of small filters ends up seeing the whole picture. Modern architectures often downsample with a stride-2 convolution instead, which learns how to summarise rather than always taking the maximum, at the cost of 73,728 parameters for a 3x3 layer from 64 to 128 channels where max pooling costs nothing.

What each depth learns#

Stacking convolutions is what turns edge detectors into object detectors, and the mechanism is the growing receptive field: how much of the original image one output unit can see. One 3x3 layer sees 3x3. Two stacked see 5x5, three see 7x7, and each pooling or stride-2 step doubles the step size so it grows much faster after that. A ResNet-18 on a 224x224 image goes through five spatial reductions, and by the last stage each unit's window covers most of the frame.

ResNet-18 at 224x224, measured by running it:

  after conv1     64 channels  112 x 112      9,408 parameters
  after maxpool   64 channels   56 x  56
  layer1          64 channels   56 x  56    147,968
  layer2         128 channels   28 x  28    525,568
  layer3         256 channels   14 x  14  2,099,712
  layer4         512 channels    7 x   7  8,393,728
  avgpool        512 channels    1 x   1
  fc                                         513,000   (1000 classes)

Total: 11,689,512 parameters.

That shape — halve the resolution, double the channels — is the standard design, and the reason is that the early layers have many positions but few distinct things to say about each one (is there an edge here, at what angle), while the late layers have few positions and a great deal to say about each (is this a wheel, a face, a signature). Probing studies and filter visualisations broadly confirm the progression from oriented edges through textures and object parts to whole objects, though it is a tendency rather than a clean division.

The practical use of this is knowing where to cut when you transfer a model. Early layers hold general edge and texture detectors that are useful on almost any image domain. Late layers hold features specific to the pretraining classes. So you keep the early layers, replace the head, and fine-tune from somewhere in the middle depending on how unlike the pretraining data your images are.

Do not train from scratch#

Almost nobody should. Take a model pretrained on a large image corpus, replace the final layer with one sized for your classes, and fine-tune. It needs a fraction of the data and compute and usually beats anything you would train from nothing.

import torchvision

num_classes = 10                    # however many classes you have
model = torchvision.models.resnet18(weights="DEFAULT")

for p in model.parameters():        # freeze the pretrained layers
    p.requires_grad = False

model.fc = nn.Linear(model.fc.in_features, num_classes)   # new head

# Train the head first, then unfreeze the top layers with a
# much lower learning rate if you need more.

Residual connections are what made depth work#

Before 2015, stacking more convolutional layers stopped helping and then started hurting, and not because of overfitting — the training loss got worse too. The fix was to add each block's input back to its output, so the block only has to learn a correction. You can watch the problem directly: build twenty blocks with and without that addition, push one backward pass through, and look at the gradient reaching the first block.

class Blk(nn.Module):
    def __init__(self, c, residual):
        super().__init__()
        self.residual = residual
        self.f = nn.Sequential(nn.Conv2d(c, c, 3, padding=1), nn.ReLU(),
                               nn.Conv2d(c, c, 3, padding=1))

    def forward(self, x):
        return torch.relu(x + self.f(x)) if self.residual else torch.relu(self.f(x))

for residual in (False, True):
    torch.manual_seed(0)
    net = nn.Sequential(*[Blk(16, residual) for _ in range(20)])
    net(torch.randn(4, 16, 16, 16)).pow(2).mean().backward()
    first = net[0].f[0].weight.grad.norm().item()
    last = net[19].f[0].weight.grad.norm().item()
    print(residual, f"{first:.3e}", f"{last:.3e}")

# False 3.543e-19 2.175e-03
# True  2.919e-01 2.105e+00

The reason is visible in the expression. Differentiating x + f(x) gives 1 + f'(x), so every block passes gradient through unchanged plus whatever its own function contributes. Twenty multiplications of numbers below one is what produced 1e-19; twenty additions of one do not shrink anything. This is why ResNet could go to 152 layers, and the same trick sits inside every transformer block you will ever read.

Have transformers replaced them#

Vision transformers cut an image into patches and run ordinary attention over them. Given enough pretraining data they match or beat convolutional networks, and they hold the top of the research benchmarks. They also carry none of the assumptions a convolution builds in — locality, and a pattern meaning the same thing anywhere — so they have to learn those from data, which is why they need much more of it before they pull ahead.

The honest production answer is that the decision is rarely about accuracy. A ViT-B/16 has 86.6 million parameters against ResNet-50's 25.6 million, and on the machine these words were written on a single 224x224 image takes 70 ms through the transformer and 23 ms through the ResNet. If the model has to run on a phone, in a camera, or on a CPU at thirty frames a second, that ratio decides it before accuracy is discussed. Attention also costs the square of the patch count, so raising the input from 224 to 512 pixels makes the attention matrix twenty-seven times larger, while a convolution's cost grows only with the pixel count.

• Reach for a convolutional backbone when data is limited, latency or power is constrained, the images are large, or the domain is unlike anything a public model was pretrained on.
• Reach for a transformer backbone when a strong pretrained checkpoint exists for something near your domain, when you need to relate distant parts of an image, or when you want text-conditioned behaviour such as searching images by description.
• In either case start from pretrained weights. The architecture argument matters far less than whether you are fine-tuning or training from nothing.

What to skip: the history of specific architectures. Knowing what AlexNet, VGG, Inception and DenseNet each contributed is pub-quiz knowledge unless you are reading papers from that decade. What actually transfers is the arithmetic above, residual connections, and knowing which backbone to pick and why.