Finding anything you can name

The detector in the recognition lesson knows 80 words. Ask it for a croissant and it has nothing to say: COCO, its training set, has donuts and cakes but no croissants. CLIP, from images and words, knows almost any word, but it describes the whole picture and cannot say where anything is.

Put the two together and you get a detector that boxes whatever you name. You type the words; it decides where each one is. This is open-vocabulary detectionOpen-vocabulary detectionObject detection where the kinds of object to find are given as text at run time, so the detector can box things named by any word or phrase rather than a fixed list of labels.Open in glossary, and the model here, OWL-ViT (Matthias Minderer and colleagues at Google, 2022), is one of the simplest ways to do it.

Every patch, scored against every word

OWL-ViT starts from CLIP: an image encoder and a text encoder trained so that pictures and their captions land close together. It changes only the end of the image side.

1. The image is stretched to 768 × 768 pixels and cut into 24 × 24 = 576 patches of 32 pixels, as in how computers see.
2. CLIP would pool the 576 output tokens into one vector for the whole picture. OWL-ViT keeps all of them. Each token gets a box, from a small network, and an embedding of what is in that box.
3. Each word you type goes through the text encoder on its own and becomes one embedding.
4. Every patch’s embedding is compared with every word’s: a cosine similarityCosine similarityA measure of how closely two vectors point in the same direction, from -1 (opposite) through 0 (perpendicular) to 1 (same direction), ignoring their lengths.Taught in Vectors and similarityOpen in glossary, adjusted by a shift and a scale that the model computes for each patch, then squashed by a sigmoidSigmoidThe S-shaped function 1 / (1 + e^-z), which squashes any number into the range 0 to 1 so it can be read as a probability.Taught in Classification and probabilityOpen in glossary into a score between 0 and 1.

So the model always makes 576 predictions, one per patch, and the words decide what they are called. Each prediction keeps the word that scores it highest, and the threshold decides which predictions you see.

Find anything you can name

Copy a linkResetFocus mode

Type words, and an open-vocabulary detector boxes whatever matches each one. No fixed list of labels.

Words to find (3 of 8)

• croissant×
• coffee cup×
• spoon×

Add a word to findAdd

Show boxes scoring at least

20%

Sample photos

Use your own photoUse the camera

Your photos and camera frames are processed on this device and never uploaded.

Recorded from OWL-ViT B/32

Loading the model's recorded outputs for the eight sample photos and the words suggested for each.

About the download

Box anything you can name in words, in any picture. This uses OWL-ViT B/32, a 153-million-parameter detector from Google (2022) built on CLIP, which draws a box around anything you name in words.

The first time, your browser downloads about 311 MB of model files from Hugging Face, plus a 14 MB runtime from this site. Both are cached for later visits. Everything runs on your device; no photo, camera frame, or text you type is sent anywhere.

Manage downloads

Checking for a saved copy

Loading the recorded results

ModelOWL-ViT B/32 (311 MB)

Boxes shownn/a

Time for this picturen/a

Challenges

0 of 3 complete

1. With the threshold at 30% or more, box both croissants on the breakfast table. The 80-label detector from the recognition lesson has no word for them. (not yet)
2. At a threshold of 10% or more, find two words the detector accepts for the same object whose scores differ by more than 15 points. (not yet)
3. Get a word to lose its best box to another word, even though it scores above the threshold there. (not yet)

What the scores mean

The two croissants score 54% and 53%. At 10%, the lower one also gets two smaller boxes over its halves and a near copy of its whole box: the partial boxes and duplicates a low threshold lets through.

Each word is scored on its own. The softmax in the recognition lesson shared one unit of probability among 1,000 labels, so a picture of nothing in particular still gave some label most of it. Here a box can score high for several words at once or for none: “elephant” never reaches 1% on the breakfast table. On the cup’s box, “coffee cup” and “cup” together score 0.93, more than any probability could.

Wording matters as much as it did for CLIP’s prompts. The same box scores 60% as “coffee cup” and 33% as “cup”; the cat on the motorcycle is 64% “cat” and 47% “kitten”. The dog that the classifier called a boxer at 77% is 20% “dog” to this model and 2% “boxer”. A score says how well the box matches this wording, so a threshold that suits one word can hide another.

Words also compete. Each prediction keeps only its best word, so with “coffee cup” and “cup” both typed, the box where “cup” scores 33% is shown as “coffee cup” at 60%. Adding a word can take boxes away from another.

How it was trained, and how it fails

This model’s encoders are OpenAI’s CLIP ViT-B/32, pretrained on 400 million pictures and captions from the web. The OWL-ViT authors added the box and embedding heads and fine-tuned everything end to end on public detection datasets of about 2 million images, such as Objects365 and Visual Genome. During fine-tuning, each image’s category names are the words, and each labeled object is matched to exactly one prediction, the training method of DETR from the recognition lesson.

To measure zero-shot detection, the authors removed every box of LVIS’s rare categories from the training data. LVIS has 1,203 categories, so any success on the rare ones comes from the image-text pretraining. In the paper, the B/32 model built on CLIP, the configuration of the model in this lab, scored 22.1 average precision on all of LVIS and 18.9 on the rare categories it had never been shown a box for. The checkpoint the authors released, which this lab runs, scores lower in their repository’s own evaluation: 16.9 on the rare categories.

The failures in the lab have the same causes as before. On the corridor, “pillar” stays under 1% though the photo is full of pillars, and the bird in the woodblock print reaches only 9%: both pictures are unlike most training photos. The cat lying across the whole frame of its photo gets 12% as “cat”, while the small cat on the motorcycle gets 64%. The model’s own card describes it as a research model, meant for studying exactly these behaviors.

import numpy as np

def unit(v):
    return v / np.linalg.norm(v, axis=-1, keepdims=True)

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

# Made-up embeddings in 3 numbers (OWL-ViT B/32 uses 512): one per image
# patch from the image encoder, one per typed word from the text encoder.
patches = unit(np.array([[0.9, 0.3, 0.1],    # a patch on a cup
                         [0.5, 0.8, 0.1],    # a patch on a croissant
                         [0.1, 0.2, 1.0]]))  # a patch on the table
words = ["coffee cup", "cup", "croissant"]
texts = unit(np.array([[1.0, 0.25, 0.0],
                       [0.9, 0.2, 0.4],
                       [0.45, 0.9, 0.0]]))

# Every patch scores every word on its own: cosine similarity, a shift and a
# scale (learned for each patch in the real model, fixed here), a sigmoid.
scores = sigmoid((patches @ texts.T - 0.85) * 20)
print("scores (rows are patches):")
print(np.round(scores, 2))
print("row sums:", np.round(scores.sum(axis=1), 2))  # not 1: no softmax

# Each patch keeps only its best word, and only above the threshold.
for p, row in enumerate(scores):
    best = row.argmax()
    if row[best] >= 0.1:
        print(f"patch {p}: {words[best]} {row[best]:.2f}")
    else:
        print(f"patch {p}: nothing above 0.10")

scores (rows are patches):
[[0.94 0.88 0.05]
 [0.06 0.03 0.94]
 [0.   0.   0.  ]]
row sums: [1.87 1.04 0.  ]
patch 0: coffee cup 0.94
patch 1: croissant 0.94
patch 2: nothing above 0.10

Sample photos, all CC0 on Wikimedia Commons: Dog by Joselodos; Cat by Vyshnavi medicharla; Cat on a motorcycle by CAPTAIN RAJU; Breakfast by Andy Li; Street by Mucklagh; Cafe by Andy Li; Corridor by Sindugab; Woodblock print by Utagawa Hiroshige (Metropolitan Museum of Art).