Vision transformers and image-text models

The architecture that took over language turned out to work on images too, once you cut an image into patches and treat them as a sequence. This section covers how that works, why it needed far more data than a convolutional network before it paid off, and how training on image and caption pairs together produces a model that can classify things nobody labelled for it. It ends on the practical question: which backbone to reach for on a given problem.

Patches as tokens#

A vision transformer cuts an image into fixed-size patches, such as 16x16 pixels, flattens each into a vector, and treats the sequence of patches like a sequence of words. A 224x224 image becomes 196 tokens. From there it is an ordinary transformer.

The patch size is the whole design decision, because it sets the sequence length, and attention costs the square of that. Halving the patch size quadruples the token count and multiplies the attention cost by sixteen. The same happens when you raise the input resolution.

16x16 patches:

  224 px ->  196 patches, attention matrix    38,416 entries      1x
  384 px ->  576 patches, attention matrix   331,776 entries    8.6x
  512 px -> 1024 patches, attention matrix 1,048,576 entries   27.3x
 1024 px -> 4096 patches, attention matrix 16,777,216 entries  436.7x

The data question#

A CNN has built-in assumptions that suit images: nearby pixels matter together, and a pattern means the same thing anywhere. A transformer has none and must learn them from data. With modest data CNNs tend to win; with very large pretraining, or strong augmentation and distillation, transformers match or beat them.

Those assumptions are called an inductive bias, and the trade is a general one: a bias that is true of your data saves you the data needed to discover it, and a bias that is false of your data costs you accuracy you cannot recover. Translation equivariance is true of natural photographs and false of, say, a fixed-camera document scan where position carries meaning, which is one of the rare cases where a transformer's lack of it is an advantage rather than a tax.

The size difference is worth having in your head. A ViT-B/16 holds 86.6 million parameters against ResNet-50's 25.6 million, and one 224x224 image took 70 ms through the transformer and 23 ms through the ResNet on the CPU these words were written on. Those exact milliseconds are a property of this machine and will not match yours; the ratio is the transferable part, and it survives onto a GPU.

Learning from captions#

CLIP-style models train an image encoder and a text encoder together on huge numbers of image and caption pairs. The objective pulls matching pairs close in a shared embedding space and pushes mismatched pairs apart. The result is an image embedding you can compare with any text.

The objective is small enough to write out, and writing it out is the fastest way to see what it does. Take a batch of image and caption pairs. Normalise both sets of vectors, take every image against every caption, and the correct answer for row i is column i — so it is an ordinary cross-entropy over the batch, computed once in each direction.

import torch
import torch.nn.functional as F

def clip_loss(img, txt, temperature=0.07):
    img = F.normalize(img, dim=1)
    txt = F.normalize(txt, dim=1)
    logits = img @ txt.T / temperature       # batch x batch similarity matrix
    target = torch.arange(len(img))          # the diagonal is the true pairing
    return (F.cross_entropy(logits, target) + F.cross_entropy(logits.T, target)) / 2

torch.manual_seed(0)
img = torch.randn(8, 32)
aligned = img + 0.1 * torch.randn(8, 32)     # captions that match their image
shuffled = aligned[torch.randperm(8)]        # right captions, wrong images
noise = torch.randn(8, 32)                   # captions with nothing to do with it

print(round(float(clip_loss(img, aligned)), 4))
print(round(float(clip_loss(img, shuffled)), 4))
print(round(float(clip_loss(img, noise)), 4))
print(round(float(torch.log(torch.tensor(8.0))), 4), "= chance for a batch of 8")

# 0.0      matched pairs, perfectly separated from the seven negatives
# 9.6629   shuffled: worse than chance, because it is confidently wrong
# 2.7001   unrelated
# 2.0794   = chance

The temperature is the other thing to understand, because it appears in every contrastive loss and is usually learned rather than fixed. Dividing the similarities by a small number before the softmax sharpens the distribution, so the loss cares intensely about the single closest negative rather than spreading its attention.

softmax over four similarity scores, same scores, three temperatures:

  temperature 1.00    0.238  0.302  0.207  0.254     nearly uniform, weak signal
  temperature 0.07    0.029  0.893  0.004  0.073     the usual setting
  temperature 0.01    0.000  1.000  0.000  0.000     saturated, gradients vanish

Classification without training#

Because image and text land in one space, you can classify into categories nobody trained on: write each class as a sentence, embed the sentences, and pick the nearest.

# encode_image and encode_text stand in for whichever CLIP implementation you
# installed; open_clip and transformers both expose methods with these names.
labels = ["cat", "dog", "invoice", "receipt"]
texts = ["a photo of a " + label for label in labels]

image_emb = encode_image(img)          # [1, d], normalised
text_emb = encode_text(texts)          # [4, d], normalised
probs = (100 * image_emb @ text_emb.T).softmax(dim=-1)
print(labels[probs.argmax()])

Zero-shot classification is a strong baseline for new categories. When you have labelled data, a linear probe on the image embeddings usually beats it cheaply: freeze the encoder, embed your training set once, and fit a logistic regression on the embeddings. It trains in seconds on a laptop, needs no gradients through the backbone, and gives you a number to beat before anyone spends a week on fine-tuning. It also fails in the obvious place — if your classes are internal product codes or defect types that no caption ever described, zero-shot has nothing to work with and the probe is the only option of the two that works at all.

Which backbone#

• Small data, edge devices, tight latency: a modern CNN.
• A pretrained transformer or CLIP-style encoder is available for your domain: start from it.
• Need to search images by text or classify open-ended categories: a CLIP-style model.
• Large images, or objects that occupy a few pixels: a CNN, or a transformer with windowed attention. Full attention over patches at high resolution is the case the arithmetic above rules out.
• You genuinely do not know: run zero-shot CLIP and a fine-tuned ResNet on the same held-out set this week. Two numbers settle it faster than any argument about inductive bias.

What to skip. The taxonomy of vision transformer variants is a research literature, not an engineering skill, and the names turn over every eighteen months. Learn patches-as-tokens, why the data requirement is higher, the quadratic cost in patch count, and the contrastive objective above. Everything else can be looked up on the day you need a specific checkpoint.