Recognizing objects with your camera

Point a phone camera at a mug and an app can say “coffee mug, 87%”. That label came from a fixed list the model was trained on, and the 87% is not what it seems. In this lesson you run two real vision models in your browser, on sample photos or on your camera, and look at exactly what they report.

Classification: one label for the whole picture

An image classifier answers one question: which label on my list best fits this picture? The list is fixed during training. For most classifiers it is the 1,000 categories of ImageNetA large collection of labeled photos organized by the WordNet hierarchy. Its best-known subset, 1.28 million training images in 1,000 categories, became the standard benchmark for image classification.Open in glossary’s classification benchmark, learned from 1.28 million labeled photos.

Those categories are oddly specific. They include more than 100 dog breeds, the moped, and the motor scooter, but no motorcycle and no general category for a person. Whatever you show the model, it must answer with one of the 1,000.

The classifier here is DeiT-Tiny, the vision transformer from the last lesson. It turns the picture into 196 patch tokens, runs 12 transformer layers, and ends with one score per label. A SoftmaxA function that turns a list of real-valued scores into probabilities that are positive and sum to 1, by exponentiating each score and dividing by the total.Taught in Predicting the next tokenOpen in glossary turns the 1,000 scores into probabilities that add up to 1. On ImageNet’s validation photos its top label is right 74.5% of the time, so it misses about one photo in four even on the kind of photos it was trained for.

Detection: what and where

Object detectionFinding every object of known kinds in an image and reporting each one as a box (its bounding box), a label, and a confidence score, instead of one label for the whole picture.Open in glossary asks more: find every object, and for each one report a box, a label, and a confidence score. Detectors are usually trained on COCO, a dataset of everyday scenes in which people outlined objects of 80 kinds: person, car, chair, cup, cat, and so on.

The detector here, YOLOS-Tiny (Fang and colleagues, 2021), reuses the same vision transformer. Next to the patch tokens it adds 100 learned detection tokens. After the last layer, each detection token outputs one box and a probability for each COCO label or “no object”. It was trained the way DETR (Carion and colleagues at Facebook AI, 2020) introduced: each real object in a training photo is matched to exactly one detection token, so a second box on the same object counts as an error. On COCO’s validation photos, YOLOS-Tiny scores 28.7 on the standard average-precision measure, against 42 for its base-sized version: a small model that misses or misplaces many small objects.

Try this

With Name the picture, try the dog, then the cat on a motorcycle. Before you look, predict what it calls the motorcycle.
Try the cafe and the corridor. How sure is the classifier, and is it right?
Switch to Find objects and pick the street. Drag the threshold down to 5%, then up to 95%, and watch boxes appear and vanish.
Try the woodblock print with both models. The detector’s 80 categories include one general “bird” label; the classifier has to choose among 1,000 photo categories, many of them individual bird species but none for birds in general.
Press Use the camera, then Analyze live, and point it at things around you. Classification updates several times a second; detection about once a second on a fast computer.

Softmax scores and detection thresholdsPython with NumPy

import numpy as np

def softmax(z):
    e = np.exp(z - z.max())
    return e / e.sum()

# A classifier's raw scores (logits) for four labels on two pictures. On
# the second, every label fits much worse (all scores 6 lower), yet the
# probabilities are identical: softmax only compares labels with each other.
good = np.array([8.0, 6.5, 4.0, 3.0])
poor = good - 6.0
print("good fit:", np.round(softmax(good), 3))
print("poor fit:", np.round(softmax(poor), 3))

def iou(a, b):
    # Overlap of two boxes (x_min, y_min, x_max, y_max) divided by their union.
    w = max(0, min(a[2], b[2]) - max(a[0], b[0]))
    h = max(0, min(a[3], b[3]) - max(a[1], b[1]))
    area = lambda r: (r[2] - r[0]) * (r[3] - r[1])
    return w * h / (area(a) + area(b) - w * h)

# A detector's output: a box, a label, and a score for each guess.
boxes = [((300, 610, 770, 950), "car", 0.99),
         ((290, 625, 775, 955), "car", 0.62),
         ((955, 585, 1045, 890), "person", 0.97),
         ((760, 470, 830, 525), "potted plant", 0.31)]
print(f"overlap of the two car boxes: {iou(boxes[0][0], boxes[1][0]):.2f}")
for t in (0.9, 0.5, 0.2):
    kept = [f"{label} {score:.2f}" for box, label, score in boxes if score >= t]
    print(f"threshold {t}: {', '.join(kept)}")

good fit: [0.801 0.179 0.015 0.005]
poor fit: [0.801 0.179 0.015 0.005]
overlap of the two car boxes: 0.91
threshold 0.9: car 0.99, person 0.97
threshold 0.5: car 0.99, car 0.62, person 0.97
threshold 0.2: car 0.99, car 0.62, person 0.97, potted plant 0.31

What the score means

The probabilities come from a softmax, which compares the labels with each other. Lower every label’s raw score by the same amount and the probabilities do not change at all. So a picture that matches none of the labels well can still give one label 78%, as the corridor does with “prison”. The score measures how strongly the model prefers that label to its other 999, not how likely it is to be right.

A model whose 80% answers are right 80% of the time is called well calibrated. Chuan Guo and colleagues showed in 2017 that modern deep networks are often poorly CalibrationHow well a model's confidence matches how often it is right. A calibrated model that says 80% is right about 80% of the time; modern neural networks are often overconfident.Open in glossary, usually overconfident, and that rescaling the scores with one fitted temperature fixes much of it.

For the detector, the slider is a cutoff on each box’s score. Lowering it finds more of the real objects but adds weak guesses and near-duplicates; raising it removes false boxes but misses real ones. This is the precision and recall trade-off from measuring a classifier, applied to every box.

Where these models fail

Every failure in the lab has a cause you can name.

The label list. A model can only say what it was trained to say. The classifier calls the motorcycle a moped because it has no motorcycle label; the detector, whose list has one, gets it right.
One label per picture. The classifier gives the cat on the motorcycle one label, and the cat loses. Busy scenes such as the breakfast table spread the probability over many labels.
Pictures unlike the training photos. The classifier calls the woodblock print an envelope, a handkerchief, or a book jacket: flat, printed objects it has seen. This is Distribution shiftWhen the data a model meets in use differs from the data it was trained on. Models often get worse under shift, sometimes without becoming any less confident.Taught in What AI gets wrongOpen in glossary, from what AI gets wrong.
Small objects. The classifier reads 224 × 224 pixels and the detector about 512 on the short side, so a cup across the room may be a few dozen pixels. Detectors score far lower on small objects: DETR, for example, averages 20.5 on COCO’s small objects and 61.1 on its large ones.

Why older detectors needed a cleanup stepOptional

Detectors before DETR, such as YOLO (Redmon and colleagues, 2016) and Faster R-CNN (Ren and colleagues, 2015), score a large number of candidate boxes, many of which cover the same object. A separate step called non-maximum suppression then keeps the highest-scoring box and deletes any other box of the same label that overlaps it by more than a set amount, measured as intersection over union: the area two boxes share divided by the area they cover together.

DETR-style training builds that rule into the model. For each training photo, the Hungarian algorithm finds the one-to-one pairing between the model’s 100 predictions and the labeled objects that costs least, and only the paired predictions are trained toward boxes. The rest are trained toward “no object”. Duplicates still appear at very low thresholds, as you can see on the cafe photo at 5%, but they score low.

Key ideas

A classifier gives a whole picture one label from a fixed list; a detector reports a box, a label, and a score for each object it finds.
Neither can name anything outside its list: ImageNet has no motorcycle, and COCO knows only 80 kinds of object.
A softmax score is the model’s preference among its own labels, which stays high even when nothing fits. Only a calibrated model’s scores read as chances of being right.
A detection threshold trades missed objects for false and duplicate boxes.
Pictures unlike the training photos, small objects, and busy scenes are where these models fail most.

Sample photos, all CC0 on Wikimedia Commons: Dog by Joselodos; Cat by Vyshnavi medicharla; Cat on a motorcycle by CAPTAIN RAJU; Breakfast by Andy Li; Street by Mucklagh; Cafe by Andy Li; Corridor by Sindugab; Woodblock print by Utagawa Hiroshige (Metropolitan Museum of Art).

Recognizing objects with your camera

In this lesson you will

Classification: one label for the whole picture

Detection: what and where