Find anything you can name

Type words, and an open-vocabulary detector boxes whatever matches each one. No fixed list of labels.

Words to find (3 of 8)

croissant
coffee cup
spoon

Show boxes scoring at least20%

Recorded from OWL-ViT B/32

Loading the model's recorded outputs for the eight sample photos and the words suggested for each.

About the download

Box anything you can name in words, in any picture. This uses OWL-ViT B/32, a 153-million-parameter detector from Google (2022) built on CLIP, which draws a box around anything you name in words.

The first time, your browser downloads about 311 MB of model files from Hugging Face, plus a 14 MB runtime from this site. Both are cached for later visits. Everything runs on your device; no photo, camera frame, or text you type is sent anywhere.

Manage downloads

Loading the recorded results

ModelOWL-ViT B/32 (311 MB)

Boxes shownn/a

Time for this picturen/a

Challenges

0 of 3 complete

With the threshold at 30% or more, box both croissants on the breakfast table. The 80-label detector from the recognition lesson has no word for them. (not yet)
At a threshold of 10% or more, find two words the detector accepts for the same object whose scores differ by more than 15 points. (not yet)
Get a word to lose its best box to another word, even though it scores above the threshold there. (not yet)

Try this

On the breakfast table, look for the croissants, then for things that are not there, such as an elephant.
Add “cup” next to “coffee cup” and compare their best scores. Where did the boxes go?
On the dog, compare “dog” with “boxer”.
Use the camera and Analyze live, then type the names of things around you. Frames are processed on your device and never uploaded.

Scoring every patch against every wordPython with NumPy

import numpy as np

def unit(v):
    return v / np.linalg.norm(v, axis=-1, keepdims=True)

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

# Made-up embeddings in 3 numbers (OWL-ViT B/32 uses 512): one per image
# patch from the image encoder, one per typed word from the text encoder.
patches = unit(np.array([[0.9, 0.3, 0.1],    # a patch on a cup
                         [0.5, 0.8, 0.1],    # a patch on a croissant
                         [0.1, 0.2, 1.0]]))  # a patch on the table
words = ["coffee cup", "cup", "croissant"]
texts = unit(np.array([[1.0, 0.25, 0.0],
                       [0.9, 0.2, 0.4],
                       [0.45, 0.9, 0.0]]))

# Every patch scores every word on its own: cosine similarity, a shift and a
# scale (learned for each patch in the real model, fixed here), a sigmoid.
scores = sigmoid((patches @ texts.T - 0.85) * 20)
print("scores (rows are patches):")
print(np.round(scores, 2))
print("row sums:", np.round(scores.sum(axis=1), 2))  # not 1: no softmax

# Each patch keeps only its best word, and only above the threshold.
for p, row in enumerate(scores):
    best = row.argmax()
    if row[best] >= 0.1:
        print(f"patch {p}: {words[best]} {row[best]:.2f}")
    else:
        print(f"patch {p}: nothing above 0.10")

scores (rows are patches):
[[0.94 0.88 0.05]
 [0.06 0.03 0.94]
 [0.   0.   0.  ]]
row sums: [1.87 1.04 0.  ]
patch 0: coffee cup 0.94
patch 1: croissant 0.94
patch 2: nothing above 0.10

Sample photos, all CC0 on Wikimedia Commons: Dog by Joselodos; Cat by Vyshnavi medicharla; Cat on a motorcycle by CAPTAIN RAJU; Breakfast by Andy Li; Street by Mucklagh; Cafe by Andy Li; Corridor by Sindugab; Woodblock print by Utagawa Hiroshige (Metropolitan Museum of Art).