Models that see
A photo is a grid of numbers. Cut it into the patches a vision transformer reads, point your camera at objects for a real classifier and detector, label your own photos with words, guess depth from a single picture, map which parts of a photo a classifier relies on, find anything you can name, cut objects out with a click, and ask a vision-language model about a photo. Every model runs on your device.
- Level
- Advanced
- Length
- 9 lessons, 2 hr 28 min
- Assumes
- The transformers track. The first lesson needs only convolutions and attention.
- Builds on
- Transformers and LLMs
- Entry point
- How computers see, an intermediate lesson that needs only Convolutions (Neural networks) and Attention, step by step (Transformers and LLMs).
- 1How computers seeEntry pointA photo reaches a model as a grid of numbers. See what a convolution's small window picks up, and how a vision transformer cuts the picture into patches and reads them as tokens.15 min
- 2Recognizing objects with your cameraPoint a real image classifier and a real object detector at sample photos or your camera. See what their labels and confidence scores mean, and where they fail.18 min
- 3Images and words togetherWith CLIP, the labels are captions you write. Score your own photos against your own words, and see why the winner is only the best of the captions you offered.16 min
- 4Depth from a single imageA model can guess how near every pixel is from one photo, using the same cues you see with one eye closed. See what its depth map does and does not tell you.16 min
- 5Where the classifier looksCover parts of a photo and run a real classifier again to map which regions its answer depends on. See what such a map reveals about a model, and what it misses.14 min
- 6Finding anything you can nameAn open-vocabulary detector boxes objects named by any words you type, by scoring every patch of the picture against each word. See what its scores mean and where it fails.16 min
- 7Cutting objects outSegmentation decides which pixels belong to an object. Click anything and SAM outlines it, or let a background remover find the main subject, and see where each one goes wrong.15 min
- 8How a model reads a pictureA vision encoder cuts the picture into patches, a projector turns each into something the language model can read, and the model answers your question about it. Run Janus-Pro and see where it goes wrong.22 min
- 9Finding a body's posePose estimation marks a body's joints. A detector boxes each person, then a pose model draws a heatmap for each of 17 keypoints and takes its peak. See what that gets right, and where it breaks.16 min
Labs in this track
- From pixels to patchesSee your photo as a vision model does, as numbers, as a convolution's small window, and as the grid of patches a vision transformer reads as tokens.
- Recognizing objectsRun a real image classifier and a real object detector on sample photos, your own pictures, or your camera, and see what their labels and scores mean.
- Labels you writeScore your own photos against labels you type, with OpenAI's CLIP running in your browser, and see how the wording and the choice of labels change the answer.
- Depth from one pictureTurn a single photo, your own picture, or a camera snapshot into a depth map and a 3D relief with Depth Anything V2 running in your browser.
- Where the classifier looksCover one square of a photo at a time and rerun a real image classifier to map which regions its answer depends on, and which it can do without.
- Find anything you can nameType any words and an open-vocabulary detector running in your browser boxes what matches each one, on sample photos, your own pictures, or the live camera.
- Cut objects outClick anything in a photo and SAM outlines it with three guesses, or let a background remover find the main subject by itself. Both run in your browser.
- How a model reads a pictureAsk Janus-Pro-1B about photos and text cards, see the 576 image tokens your picture becomes, and check how sure the model was of every word, right or wrong.
- Find a body's keypointsA person detector boxes each person, then ViTPose+ finds 17 keypoints in each box, from the nose to the ankles. See the heatmap behind every keypoint, and where pose estimation fails.