A detector draws a box around the dog. The box also holds a lot of lawn. To cut the dog out of the photo, measure how much of the picture it takes up, or tell a robot arm exactly where its edges are, you need to know which pixels are dog. That is Image segmentationDeciding which pixels of a picture belong to which object or region, producing a mask for each, rather than a label for the whole picture or a box around each object.Open in glossary: a mask, one yes or no per pixel, instead of a label or a box.
Classic segmentation models, like classic detectors, learn a fixed list of classes and paint every pixel with one of them. This lesson covers two newer kinds that need no list: one outlines whatever you point at, and one finds the main subject by itself.
Click, and get three guesses
The Segment Anything Model, SAM (Alexander Kirillov and colleagues at Meta, 2023), does Promptable segmentationSegmentation guided by a prompt such as a click, a box, or a rough mask, so one model can outline whatever object the person points at, without a fixed list of classes. SAM is the best-known example.Open in glossary: you give it a click, a box, or a rough mask, and it outlines the object you meant. Its creators built a dataset of over a billion masks on 11 million licensed photos, SA-1B, in a loop where the model proposed masks, people corrected them, and the model was retrained on the corrections; in the last round the model made all the masks on its own.
A single click is ambiguous. A click on a dog’s collar could mean the collar, the dog’s neck, or the whole dog. Trained to give one answer, a model averages the possibilities into a blur. So SAM gives three masks, which the authors found covers most cases (nested masks are rarely more than three deep: whole, part, and subpart), and it predicts each mask’s quality as an Intersection over unionHow well two regions agree: the area they share divided by the area either one covers, from 0 (no overlap) to 1 (identical). Used to compare boxes in detection and masks in segmentation.Open in glossary with the true object, which it uses to rank them.
SAM is built for clicking. A large image encoder reads the picture once. A small prompt encoder and mask decoder then turn each set of clicks into masks; the SAM authors ran them in a web browser on a CPU in about 50 milliseconds. The lab uses SAM 2.1 tiny (Nikhila Ravi and colleagues at Meta, 2024), a 39-million-parameter version of SAM’s successor, which also tracks objects through video.
Cut objects out
SAM outlines whatever you click, offering three guesses. A background remover finds the main subject by itself.
Your photos and camera frames are processed on this device and never uploaded.
Loading the recorded results
SAM 2.1 tiny reads each picture once (the slow part) and then outlines every click in a fraction of a second. BiRefNet lite needs no clicks.
Challenges
0 of 4 complete
- With a single click, get three candidate masks where the largest is at least ten times the size of the smallest. (not yet)
- Show a mask that fills less than half of its own bounding box. (not yet)
- Find a click where SAM rates even its best mask below 50%. (not yet)
- Find a photo where the subject finder keeps nothing at all. (not yet)
Try this
- On the dog, click the collar, then the head, then the lawn. Before each click, predict how different the three guesses will be.
- Pick SAM’s largest guess for the dog and read how much of its own box it fills.
- Try the marked pillar in the corridor and the plum branch in the print. How highly does SAM rate its own best mask?
- Switch to Find the subject and go through the sample photos. Predict which ones will come back empty.
- With the model downloaded, add a few clicks, then switch to Remove and click a part you do not want.
What the masks show
The collar click shows the ambiguity at its widest: the guesses run from a strip of the collar, 0.1% of the picture, through the whole collar, 0.5% and SAM’s pick, to the whole dog, 22%. Which one you wanted is something only you know, which is why SAM shows all three.
A mask is a different kind of answer from a box. The whole dog fills only 44% of its own bounding box, because the box also holds the lawn between its legs and around its tail. The bird in the woodblock print, with its long tail, fills 18% of its box.
SAM’s ratings are useful, not perfect. It rates its own best guess low on thin, tangled, or repeated things: 20% for one plum branch among many, 34% for one pillar in a row of identical pillars. A low rating is a signal to add another click.
Remove clicks tell SAM to leave a region out, which is how you correct a guess that spills into something next to the object. They do not always take. In a case measured for this lesson, a click on the dog’s body plus a remove click on its head, the whole-dog mask went from 22.30% of the picture to 22.24%: SAM kept the head as part of the dog. One case is not a rule, so with the model downloaded, try others.
SAM never says what it outlined. It has no labels at all, so to cut out every cup in a photo you pair it with a detector that names things, such as the open-vocabulary detector: the detector finds the cups, and each of its boxes becomes SAM’s prompt.
Finding the subject on its own
Background removal asks a different question: what is this photo of? The model in the lab is BiRefNet lite (Peng Zheng and colleagues, 2024). It was trained on public datasets of photos paired with hand-drawn masks of their foreground, from fine-detail collections such as DIS5K to salient-object sets such as DUTS, and it outputs an Alpha matteA per-pixel opacity map from 0 (background) to 1 (subject), with values in between at soft edges such as hair, used to cut a subject out and place it over a new background.Open in glossary: a value from 0 to 255 per pixel, soft at edges such as fur.
It decides what counts as the subject. The dog and the motorcycle come out cleanly. On the breakfast table it keeps the cups, glasses, and croissants together with their plates and napkins. The street, the cafe, and the corridor come back empty: these photos have no single main subject, and the model was trained on photos that do.
Masks, IoU, and a soft cut-outPython with NumPy
import numpy as np
# The true outline of an L-shaped object in a 6 x 8 picture (True = object).
truth = np.zeros((6, 8), dtype=bool)
truth[1:5, 2] = True # the upright
truth[4, 2:7] = True # the foot
def iou(a, b):
"""Intersection over union: overlap divided by the area either one covers."""
return (a & b).sum() / (a | b).sum()
# Two guesses: just the upright, and the whole L plus one stray pixel.
upright = np.zeros_like(truth)
upright[1:5, 2] = True
whole = truth.copy()
whole[0, 7] = True
print(f"upright: IoU {iou(upright, truth):.2f}")
print(f"whole: IoU {iou(whole, truth):.2f}") # SAM predicts this number for each guess
# How much of its own bounding box the object fills.
rows, cols = np.nonzero(truth)
box = (rows.max() - rows.min() + 1) * (cols.max() - cols.min() + 1)
print(f"the L fills {truth.sum() / box:.0%} of its box")
# Cutting out with a soft matte: each pixel mixes the photo and a new backdrop.
alpha = np.array([0.0, 0.3, 1.0]) # background, a wisp of hair, subject
photo, backdrop = 200.0, 50.0 # brightness of one photo pixel and the backdrop
print("composite:", alpha * photo + (1 - alpha) * backdrop)upright: IoU 0.50
whole: IoU 0.89
the L fills 40% of its box
composite: [ 50. 95. 200.]Why this lab reads pictures at 512 pixels, and other limitsOptional
BiRefNet was trained on pictures at 1,024 × 1,024 pixels. At that size, the ONNX export on the Hugging Face Hub needs more working memory than WebAssembly can address in a browser (4 GB at most), and it fails with an out-of-memory error at both 32-bit and 16-bit precision. The lab uses an export of the same weights made by a third party at 512 × 512, which needs a quarter of the activations. We checked it against the official PyTorch model at 512 pixels: the two agree to within 0.2% to 2% of the alpha range on average, from differences in image resizing. Reading at half its training size costs quality on some photos; at 1,024 pixels, run outside the browser, the same model keeps the bird and branch of the woodblock print and drops almost all of the paper.
SAM 2.1 reads every picture at 1,024 × 1,024 pixels, so a 4:3 photo is stretched, and the masks come back from a 256 × 256 grid scaled up, which rounds fine detail. The SAM 2 authors report that it is more accurate than the original SAM on images and six times faster, and its training data adds SA-V, 35.5 million masks across 50,900 videos.
Key ideas
- Segmentation marks which pixels belong to an object; a box only says roughly where it is.
- One click is ambiguous, so SAM returns three nested guesses and rates each with a predicted intersection over union.
- SAM reads the picture once with a large encoder and answers each click with a small, fast decoder, which makes clicking interactive.
- SAM outlines anything but names nothing; pair it with a detector to find and cut out named things.
- A background remover decides for itself what the subject is, from patterns in its training photos, and can come back empty when a photo has no single subject.
Sample photos, all CC0 on Wikimedia Commons: Dog by Joselodos; Cat by Vyshnavi medicharla; Cat on a motorcycle by CAPTAIN RAJU; Breakfast by Andy Li; Street by Mucklagh; Cafe by Andy Li; Corridor by Sindugab; Woodblock print by Utagawa Hiroshige (Metropolitan Museum of Art).