Labels you write

CLIP embeds your photo and each label's caption in one space; the caption closest to the photo wins.

Dimmed: the edges CLIP's preprocessing crops away. It reads only the middle square.

Wrap each label as “a photo of a …”

Recorded from CLIP ViT-B/32

Loading the model's recorded outputs for the eight sample photos and 43 labels, suggested as you type.

About the download

Score any photo against labels you write. This uses CLIP ViT-B/32, a 2021 model from OpenAI that maps images and text into one shared space of 512 numbers, so a photo and its description land close together.

The first time, your browser downloads about 306 MB of model files from Hugging Face, plus a 14 MB runtime from this site. Both are cached for later visits. Everything runs on your device; no photo, camera frame, or text you type is sent anywhere.

Manage downloads

Loading the recorded results

Each embedding512 numbers, length 1

Logit scale (learned)100 × cosine

Best captionn/a

Challenges

0 of 3 complete

Remove labels until only one is left. How sure is CLIP now? (not yet)
On the dog photo, keep the label dog and add a label that beats it. (not yet)
On the woodblock print, make a label about how the picture was made beat a label about the bird. (not yet)

Try this

Add boxer to the dog photo’s labels. Then remove labels one at a time.
On the woodblock print, compare bird with woodblock print.
Turn the “a photo of a” wrapper off and on.
Try your own photo or the camera with labels of your own. Nothing leaves your device. CLIP’s own model card warns against using it to label people.

Zero-shot scores from cosinesPython with NumPy

import numpy as np

def zero_shot(cosines, scale=100.0):
    # CLIP multiplies each cosine by its learned scale (100), then takes a
    # softmax over the captions you offered, and only those.
    logits = scale * np.asarray(cosines)
    p = np.exp(logits - logits.max())
    return p / p.sum()

# Cosines CLIP ViT-B/32 gives the lab's dog photo for "a photo of a ..."
cos = {"dog": 0.2806, "cat": 0.2090, "car": 0.1828, "boxer": 0.3121}
for labels in (["dog", "cat", "car"], ["dog", "cat", "car", "boxer"], ["car"]):
    p = zero_shot([cos[l] for l in labels])
    print(", ".join(f"{l} {x:.1%}" for l, x in zip(labels, p)))

dog 99.9%, cat 0.1%, car 0.0%
dog 4.1%, cat 0.0%, car 0.0%, boxer 95.9%
car 100.0%

Sample photos, all CC0 on Wikimedia Commons: Dog by Joselodos; Cat by Vyshnavi medicharla; Cat on a motorcycle by CAPTAIN RAJU; Breakfast by Andy Li; Street by Mucklagh; Cafe by Andy Li; Corridor by Sindugab; Woodblock print by Utagawa Hiroshige (Metropolitan Museum of Art).