Asked about Vision transformers and image-text models

How does CLIP enable zero-shot classification?

An answer that works

It learns a shared embedding space for images and text by training on image and caption pairs with a contrastive objective. To classify, embed the image and a text prompt for each class, and pick the class whose text embedding is most similar to the image.

Then they push further

When would a linear probe beat zero-shot?

When you have even a modest labelled set, or when your classes are described poorly by short text, such as internal product codes. The probe learns directly from your examples.

Reading an answer is not being able to give one. The section that teaches this is Vision transformers and image-text models, 3 hours of Convolutional networks.

Learn this properly

Also asked about Convolutional networks

Why are convolutions suited to images?Would you train a vision model from scratch?Have vision transformers replaced convolutional networks?What is the difference between detection and segmentation?How do you choose data augmentations?What is the output shape of a 3x3 convolution with stride 2 and padding 1 on a 64x64x32 input with 64 filters, and how many parameters does it have?Explain mean average precision for object detection.