Recognizing objects

Run a real image classifier and a real object detector on sample photos, your own pictures, or your camera, and see what their labels and scores mean.

IntermediateExplained in Recognizing objects with your camera

Recognizing objects

A classifier names the whole picture with one of 1,000 labels; a detector boxes every object it knows from a list of 80.

Model
Sample photos

Your photos and camera frames are processed on this device and never uploaded.

Recorded from DeiT-Tiny

Loading the model's recorded outputs for the eight sample photos.

About the download

Name any picture with one of ImageNet's 1,000 categories. This uses DeiT-Tiny, a 6-million-parameter vision transformer from Meta (2020) that sorts a photo into 1,000 ImageNet categories.

The first time, your browser downloads about 24 MB of model files from Hugging Face, plus a 14 MB runtime from this site. Both are cached for later visits. Everything runs on your device; no photo or camera frame is sent anywhere.

Manage downloads

Loading the recorded results

ModelDeiT-Tiny (24 MB)
Time for this picturen/a
Entropy over all 1,000 labelsn/a

Challenges

0 of 3 complete

  1. Name the cat on a motorcycle with the classifier, then use the detector to find both the cat and the motorcycle in it. (not yet)
  2. Find a picture whose top label gets less than 30%. (not yet)
  3. Find a picture the classifier labels wrongly while giving that label more than 70%, and mark it wrong. (not yet)

Try this

  • Classify the cat on a motorcycle, then detect objects in it. What does each model see?
  • With Find objects, slide the threshold from 5% to 95% on the cafe photo.
  • Find a picture where the classifier is confident and wrong.
  • Use the camera and Analyze live. Frames are processed on your device and never uploaded.
Softmax scores and detection thresholdsPython with NumPy
import numpy as np

def softmax(z):
    e = np.exp(z - z.max())
    return e / e.sum()

# On the second picture every label fits much worse (all scores 6 lower),
# yet the probabilities are identical: softmax only compares labels.
good = np.array([8.0, 6.5, 4.0, 3.0])
print("good fit:", np.round(softmax(good), 3))
print("poor fit:", np.round(softmax(good - 6.0), 3))

# A detector's guesses: label and score. The threshold decides which to show.
guesses = [("car", 0.99), ("car", 0.62), ("person", 0.97), ("potted plant", 0.31)]
for t in (0.9, 0.5, 0.2):
    print(f"threshold {t}:", ", ".join(f"{l} {s:.2f}" for l, s in guesses if s >= t))
good fit: [0.801 0.179 0.015 0.005]
poor fit: [0.801 0.179 0.015 0.005]
threshold 0.9: car 0.99, person 0.97
threshold 0.5: car 0.99, car 0.62, person 0.97
threshold 0.2: car 0.99, car 0.62, person 0.97, potted plant 0.31

Sample photos, all CC0 on Wikimedia Commons: Dog by Joselodos; Cat by Vyshnavi medicharla; Cat on a motorcycle by CAPTAIN RAJU; Breakfast by Andy Li; Street by Mucklagh; Cafe by Andy Li; Corridor by Sindugab; Woodblock print by Utagawa Hiroshige (Metropolitan Museum of Art).

Try "embedding", "softmax", "overfitting", or "backpropagation".