Beyond classification

Classification tells you what is in an image. Most real vision work needs more than that: where each object is, which pixels belong to it, how many there are. This section covers detection and segmentation, the augmentation that stands in for data you do not have, and the metrics these tasks use, which are not accuracy and take some getting used to. It is also where you learn to choose between a fast one-stage detector and a more accurate two-stage one, which is the practical decision most of the time.

Which task do you actually have#

Start here, because the answer changes the labelling budget by an order of magnitude and that is usually the binding constraint. Read the business question and match it to the cheapest task that answers it.

the question being asked                        the task            label cost

"is there a defect in this image"               classification      one click
"how many pallets are on the lorry"             detection           a box each
"which bin should this parcel go in"            classification      one click
"how far is the pedestrian from the kerb"       detection           a box each
"what fraction of the field is diseased"        semantic segmentation  a mask
"how many leaves, and how big is each"          instance segmentation  a mask each
"read the serial number"                        detection + OCR     a box each

Detection#

Classification says what is in an image. Detection says where each object is, with a box and a label. One-stage detectors predict boxes directly and are fast; two-stage detectors propose regions then classify them and are more accurate. For most applications a one-stage detector is the practical default.

The mechanism worth understanding, because it is where the surprises live, is that every detector predicts far more boxes than there are objects. A one-stage detector emits a prediction at every position of several feature maps — tens of thousands of candidates for an image containing three things — and then a filtering step keeps the best and discards the duplicates. That step is non-maximum suppression: sort by confidence, keep the top box, throw away anything overlapping it by more than a threshold, repeat.

import torch
from torchvision.ops import nms, box_iou

boxes = torch.tensor([[10., 10., 50., 50.],      # three near-duplicate boxes
                      [12., 12., 52., 52.],      # on the same object
                      [14.,  8., 54., 48.],
                      [200., 200., 240., 240.],  # and two on a second object
                      [205., 205., 245., 245.]])
scores = torch.tensor([0.95, 0.90, 0.85, 0.80, 0.75])

print((box_iou(boxes, boxes) * 100).round().to(int).tolist())
for threshold in (0.9, 0.5, 0.3):
    print(threshold, nms(boxes, scores, threshold).tolist())

# [[100, 82, 75, 0, 0], [82, 100, 75, 0, 0], [75, 75, 100, 0, 0],
#  [0, 0, 0, 100, 62], [0, 0, 0, 62, 100]]
# 0.9 [0, 1, 2, 3, 4]
# 0.5 [0, 3]
# 0.3 [0, 3]

The other thing to know before picking an architecture is that one-stage and two-stage is less of a divide than it used to be. The practical ordering is: a modern one-stage detector for anything real-time or on a device; a two-stage detector when small objects or tight boxes matter more than latency; and an open-vocabulary detector, which takes the class names as text, when your classes change often or you have almost no labelled data.

Segmentation#

Assign a label to every pixel. Semantic segmentation labels pixels by class; instance segmentation also separates individual objects of the same class. Architectures typically downsample to understand the scene, then upsample back to full resolution, with skip connections to recover fine detail.

The skip connections are not an optimisation, they are the reason the output has edges. Downsampling from 512 to 16 pixels across is what lets the network know it is looking at a road rather than grey texture, and it also destroys every boundary. Upsampling alone gives you a blurred blob in roughly the right place. Copying the high-resolution feature maps from the encoder across to the matching decoder stage is what puts the boundary back, and a U-shaped architecture with those copies is still the default for medical and satellite imagery two decades on.

Segmentation is scored differently from classification, and using accuracy here is the standard mistake. If 2 per cent of pixels are tumour, a model predicting no tumour anywhere scores 98 per cent pixel accuracy. The metrics that mean something are IoU per class, also called the Jaccard index, averaged over classes; and the Dice coefficient, which is twice the intersection over the sum of the two areas. Dice is more forgiving on small regions, which is why medical work tends to report it and why you should never compare a Dice number with an IoU number.

Augmentation#

Create new training examples by transforming existing ones: flips, crops, rotations, colour changes. It is one of the most effective regularisers in vision and nearly free. The rule is to only apply transformations that preserve the label.

The size of the effect is easy to underestimate, so here it is measured. The task is two shapes of identical area, a 4x4 square and a 2x8 bar. The training set has every object near the centre of the frame, which is what a dataset collected by one person with one camera setup looks like. The test set has them anywhere.

import torch
import torch.nn as nn

def make(n, centred, seed):
    g = torch.Generator().manual_seed(seed)
    X = torch.zeros(n, 1, 32, 32); y = torch.zeros(n, dtype=torch.long)
    for i in range(n):
        cls = int(torch.randint(0, 2, (1,), generator=g)); y[i] = cls
        lo, hi = (12, 16) if centred else (2, 22)
        r = int(torch.randint(lo, hi, (1,), generator=g))
        c = int(torch.randint(lo, hi, (1,), generator=g))
        if cls == 0: X[i, 0, r:r+4, c:c+4] = 1.0      # square, 16 pixels
        else:        X[i, 0, r:r+2, c:c+8] = 1.0      # bar, also 16 pixels
    return X, y

def shift(X, g):                                      # the augmentation
    out = torch.zeros_like(X)
    for i in range(len(X)):
        dy = int(torch.randint(-10, 11, (1,), generator=g))
        dx = int(torch.randint(-10, 11, (1,), generator=g))
        out[i] = torch.roll(X[i], shifts=(dy, dx), dims=(1, 2))
    return out

Xtr, ytr = make(400, True, 0)          # training: always near the centre
Xte, yte = make(400, False, 1)         # test: anywhere in the frame

def train(augment):
    torch.manual_seed(0)
    m = nn.Sequential(nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
                      nn.Conv2d(8, 16, 3, padding=1), nn.ReLU(),
                      nn.Flatten(), nn.Linear(16 * 16 * 16, 2))
    opt = torch.optim.Adam(m.parameters(), lr=1e-3)
    g = torch.Generator().manual_seed(1)
    for _ in range(300):
        xb = shift(Xtr, g) if augment else Xtr
        opt.zero_grad()
        nn.functional.cross_entropy(m(xb), ytr).backward()
        opt.step()
    with torch.no_grad():
        return ((m(Xtr).argmax(1) == ytr).float().mean().item(),
                (m(Xte).argmax(1) == yte).float().mean().item())

for augment in (False, True):
    print(augment, ["%.3f" % v for v in train(augment)])

# False ['1.000', '0.560']     perfect on training, barely above chance elsewhere
# True  ['1.000', '1.000']

Note also what the fix is not. A convolution is translation-equivariant, so the feature maps move correctly with the object; but the flatten-and-linear head at the end reads absolute positions, and that is what memorised the centre. Replacing the head with global average pooling would have helped for the same reason augmentation did. Augmentation is the general tool because it works whatever the architecture got wrong.

Evaluating detection#

Detection uses intersection over union to decide whether a predicted box matches a real one, and mean average precision across classes and thresholds as the summary. As with any metric, check it against the errors that actually matter in the application.

Two habits make that check real rather than rhetorical. Break the metric down by object size — COCO reports small, medium and large separately for exactly this reason, and a system whose overall mAP is 0.6 can be at 0.15 on the small objects that are the point of the project. And decide up front which error is worse, because average precision summarises the whole confidence curve and you will ship a single threshold. A missed defect and a false alarm are not the same cost, and the threshold, not the model, is where you express that.

What to skip. The architectural details of any particular detector family — anchor box design, feature pyramid arithmetic, the exact assignment rules — are things you read when you have a specific problem the defaults do not solve. What earns its place up front is choosing the right task, understanding the candidate-plus-suppression shape, knowing that segmentation is scored by IoU or Dice and never by accuracy, and treating augmentation as a first-class part of the model rather than a finishing touch.