Images and words together

The classifier in the last lesson could only answer with one of its 1,000 ImageNet labels. If you want to sort your photos into “my bike” and “not my bike”, or “sunny” and “rainy”, you would have to collect labeled examples and train it again.

CLIP (Radford and colleagues at OpenAI, 2021) removes that step: you write the labels yourself. The frontiers lesson on CLIP trains a miniature version and shows how its two encoders learn one shared space. Here you use the real model, on your own photos and words.

Captions as labels

CLIP has two encoders. The image encoder here, ViT-B/32, is a vision transformer with 32-pixel patches: it reads a 224 × 224 square as 49 patch tokens, the setting from the first lesson’s challenge. The text encoder is a transformer over the caption’s tokens. Each turns its input into an EmbeddingA list of numbers (a vector) that represents a token, word, sentence, or image, learned so that similar things end up with similar vectors.Taught in Word embeddingsOpen in glossary of 512 numbers, scaled to length 1. Both were trained on 400 million image and caption pairs from the internet so that a picture and its caption land close together.

To classify a photo, CLIP does three things:

Turn each label into a caption, such as “a photo of a dog”, and embed it.
Compute the Cosine similarityA measure of how closely two vectors point in the same direction, from -1 (opposite) through 0 (perpendicular) to 1 (same direction), ignoring their lengths.Taught in Vectors and similarityOpen in glossary between the photo’s embedding and each caption’s.
Multiply each cosine by 100, CLIP’s learned scale, and take a SoftmaxA function that turns a list of real-valued scores into probabilities that are positive and sum to 1, by exponentiating each score and dividing by the total.Taught in Predicting the next tokenOpen in glossary over the captions.

The caption that points in the most similar direction wins. Nothing in CLIP was trained on your categories, which is why this is called Zero-shotDoing a task without any examples of that specific task during training or in the prompt, for example classifying images into categories the model was never trained to label.Taught in One space for images and textOpen in glossary classification.

Labels you write

CLIP embeds your photo and each label's caption in one space; the caption closest to the photo wins.

Dimmed: the edges CLIP's preprocessing crops away. It reads only the middle square.

Wrap each label as “a photo of a …”

Recorded from CLIP ViT-B/32

Loading the model's recorded outputs for the eight sample photos and 43 labels, suggested as you type.

About the download

Score any photo against labels you write. This uses CLIP ViT-B/32, a 2021 model from OpenAI that maps images and text into one shared space of 512 numbers, so a photo and its description land close together.

The first time, your browser downloads about 306 MB of model files from Hugging Face, plus a 14 MB runtime from this site. Both are cached for later visits. Everything runs on your device; no photo, camera frame, or text you type is sent anywhere.

Manage downloads

Loading the recorded results

Each embedding512 numbers, length 1

Logit scale (learned)100 × cosine

Best captionn/a

Challenges

0 of 3 complete

Remove labels until only one is left. How sure is CLIP now? (not yet)
On the dog photo, keep the label dog and add a label that beats it. (not yet)
On the woodblock print, make a label about how the picture was made beat a label about the bird. (not yet)

Try this

With the dog photo, add boxer. Predict first: does the general label or the specific one win?
Pick the woodblock print, then try bird and woodblock print as labels.
Turn off the “a photo of a” wrapper and watch every cosine move.
Pick the cat on a motorcycle with labels cat and motorcycle. Which does CLIP think the photo is about?
With the camera, write a word on paper and hold it next to an object. Does the written word pull the caption with the same word toward the top?

Zero-shot scores from cosinesPython with NumPy

import numpy as np

def unit(v):
    return v / np.linalg.norm(v)

# Cosine similarity is the dot product of two unit-length vectors.
a, b = unit(np.array([3.0, 4.0])), unit(np.array([4.0, 3.0]))
print(f"cosine of two 2-number vectors: {a @ b:.2f}")

def zero_shot(cosines, scale=100.0):
    # CLIP multiplies each cosine by its learned scale (100), then takes a
    # softmax over the captions you offered, and only those.
    logits = scale * np.asarray(cosines)
    p = np.exp(logits - logits.max())
    return p / p.sum()

# Cosines CLIP ViT-B/32 gives the lab's dog photo for "a photo of a ..."
# captions (measured in the browser; its embeddings have 512 numbers).
cos = {"dog": 0.2806, "cat": 0.2090, "car": 0.1828, "boxer": 0.3121}
for labels in (["dog", "cat", "car"], ["dog", "cat", "car", "boxer"], ["car"]):
    p = zero_shot([cos[l] for l in labels])
    print(", ".join(f"{l} {x:.1%}" for l, x in zip(labels, p)))

gap = cos["boxer"] - cos["dog"]
print(f"odds boxer to dog: e^(100 x {gap:.4f}) = {np.exp(100 * gap):.1f}")

cosine of two 2-number vectors: 0.96
dog 99.9%, cat 0.1%, car 0.0%
dog 4.1%, cat 0.0%, car 0.0%, boxer 95.9%
car 100.0%
odds boxer to dog: e^(100 x 0.0315) = 23.3

Small cosines, big probabilities

The cosines are small. In the lab, a caption that fits gets about 0.3 and one that does not about 0.2, and the gap between the two best captions is often a few hundredths. CLIP’s scale of 100 stretches those gaps: a difference of 0.03 becomes 3 in the softmax, which is odds of $e^3$ , about 20 to 1. That is why the dog photo goes from 99.9% “dog” to 95.9% “boxer” when you add one label, even though the cosine for “dog” did not change at all.

From cosines to probabilities

For an image embedding $\mathbf{v}$ and caption embeddings $\mathbf{t}_1, \dots, \mathbf{t}_K$ , all of length 1, the probability of caption $k$ is

p_k = \frac{e^{s\, \mathbf{v}\cdot\mathbf{t}_k}}{\sum_{j=1}^{K} e^{s\, \mathbf{v}\cdot\mathbf{t}_j}}, \qquad s = 100.

$s$ is $1/\tau$ for the temperature $\tau$ in the frontiers lesson. CLIP learned it during training, starting from $1/0.07 \approx 14.3$ , and the training clipped it at 100, which is where it ended up. The odds between two captions are $p_a / p_b = e^{s(\mathbf{v}\cdot\mathbf{t}_a - \mathbf{v}\cdot\mathbf{t}_b)}$ , so they depend only on the gap between their cosines.

The wording matters

CLIP learned from captions, so labels work best when they read like captions. The CLIP paper found that the template “A photo of a {label}.” added 1.3 points of ImageNet accuracy over bare labels, and that averaging the embeddings of 80 different templates (“a sketch of a …”, “a photo of a large …”) added 3.5 more. A bare word can also be ambiguous: “boxer” might be a dog or an athlete, and “crane” a bird or a machine.

The captions also capture more than the subject. The woodblock print matches “woodblock print” better than “bird”, because captions for such pictures often name the medium. The cat on a motorcycle matches “motorcycle” better than “cat”, because the motorcycle fills the frame. Note too that CLIP’s preprocessing keeps only the middle square of the photo, so whatever sits near the long edges is cut off before the model sees it.

Where it fails

Only your labels. It cannot say “none of these”. A low best cosine is the only hint.
Counting and relations. As the frontiers lesson describes, matching a bag of words is often enough to win during training, so “two cats” against “three cats”, or “a cat on a motorcycle” against “a motorcycle on a cat”, can come out close.
Written words. In 2021, OpenAI researchers (Goh and colleagues) showed that attaching a handwritten note reading “iPod” to an apple made CLIP classify the apple as an iPod. They called these typographic attacks: the model links the written word to the concept so strongly that the text in the photo can outweigh the object.
Biases from the internet. The training captions came from the internet, and so do their associations. OpenAI’s model card for CLIP reports significant race and gender disparities when photos of people were scored against crime-related and animal labels. It calls CLIP a research output, considers any deployed use out of scope, advises against even non-deployed uses, such as image search in a constrained setting, without thorough testing on a fixed set of labels, and rules out surveillance and facial recognition entirely. This lab is for studying exactly that behavior; do not use it to label people.

Key ideas

CLIP classifies by comparing a photo’s embedding with the embeddings of captions you write; the closest caption wins.
Probabilities are a softmax over your labels only, after multiplying cosines by a learned scale of 100, so small cosine gaps become large probability gaps.
Adding, removing, or rewording a label can change the winner; labels written as captions work better than bare words.
The shared space encodes medium, style, and written words as well as subject, which helps and also opens failures such as typographic attacks.
No probability here says whether any label is true; at most the cosines hint that nothing fits.

Sample photos, all CC0 on Wikimedia Commons: Dog by Joselodos; Cat by Vyshnavi medicharla; Cat on a motorcycle by CAPTAIN RAJU; Breakfast by Andy Li; Street by Mucklagh; Cafe by Andy Li; Corridor by Sindugab; Woodblock print by Utagawa Hiroshige (Metropolitan Museum of Art).