A fitness app counts your squats. A video game character copies your arms. A coach compares an athlete’s knee angle across a season. Each of these starts with the same step: find where the body’s joints are in a picture. That is Pose estimationFinding where a body's joints and other landmarks (keypoints such as the shoulders, elbows, knees, and ankles) are in a picture, usually drawn as a stick-figure skeleton.Open in glossary, and its answer is usually drawn as a stick figure.
Seventeen points per person
A pose model does not outline the body. It reports a short list of KeypointA named landmark on a body or object, such as the left elbow or the nose, given as a point in the picture. The COCO dataset labels 17 for each person, from the nose to the ankles.Open in glossary, each a point in the picture with a name. The most common list comes from the COCO dataset, which has about 250,000 people labeled this way: 17 keypoints, the nose, both eyes, both ears, and both shoulders, elbows, wrists, hips, knees, and ankles. Left and right are the person’s own, so on someone facing you, their left wrist is on your right.
COCO’s labelers marked each keypoint as visible, as present but hidden (behind an arm, under a sleeve), or as not labeled at all. Hidden keypoints still get a position, so a model trained on COCO learns to guess where a hidden elbow probably is. Keep that in mind: it explains several of the failures below.
One box at a time
A photo can hold any number of people. There are two ways to deal with that.
A Top-down pose estimationFinding body poses in two steps. A detector first draws a box around each person, then a pose model finds the keypoints of the one person in each box. Its cost grows with the number of people.Open in glossary system first runs a person detector, like the one in Recognizing objects with your camera, which draws a box around each person. Then a pose model reads one box at a time and returns the 17 keypoints of the one person in it. The lab works this way: YOLOS-Tiny finds the people, and ViTPose+ small reads each box. The cost grows with the crowd, one pass of the pose model per person, and a person the detector misses gets no pose at all.
A Bottom-up pose estimationFinding body poses by detecting every keypoint of every person in the whole picture at once, then grouping the keypoints into people, as OpenPose does with part affinity fields.Open in glossary system reads the whole picture once and finds every keypoint of every person, then works out which keypoints belong together. OpenPose (Zhe Cao and colleagues at Carnegie Mellon, 2017) does it with part affinity fields: alongside each joint’s map, the network predicts, for every limb, which way the limb points at each pixel, and a fast matching step joins elbows to the wrists their forearms lead to. Its authors report real-time speed irrespective of the number of people, and point out the top-down weakness: if the detector fails, there is no way to recover.
A heatmap for every keypoint
ViTPose (Yufei Xu and colleagues at the University of Sydney, 2022) is a plain Vision transformerA transformer that reads an image by cutting it into square patches, turning each patch into a token vector, and attending across all the patches at once. Often shortened to ViT.Taught in How computers seeOpen in glossary, the kind from How computers see, with a small decoder on top. It reads each box as a 192 by 256 pixel crop: 12 by 16 patches of 16 pixels. The decoder enlarges the transformer’s output four times and returns one Keypoint heatmapA small grid a pose model outputs for each keypoint, with a high value wherever it thinks that keypoint is. It is trained to draw a bump centered on the true position, and the keypoint is read from the grid's peak.Open in glossary per keypoint, a grid of 48 by 64 cells, each covering 4 by 4 pixels of the crop.
During training, the target for each keypoint was a heatmap with a small round bump, a two-dimensional Gaussian, centered on where the labeler put the keypoint, and the model learned to match it, cell by cell (mean squared error, as in earlier heatmap models such as SimpleBaseline, Bin Xiao and colleagues, 2018). Reading a keypoint is the reverse: find the highest cell, refine that position to a fraction of a cell from the shape of the bump around it, and map it back from the crop to the photo. The height of the peak is the keypoint’s score.
Find a body's keypoints
A detector boxes each person, then a pose model finds 17 keypoints inside each box.
Your photos and camera frames are processed on this device and never uploaded.
left side right side nose The person's own left and right. Tap a box to choose a person.
1. Find each person YOLOS-Tiny
Loading the recorded results
2. Find 17 keypoints in each box ViTPose+ small
Loading the recorded results
Challenges
0 of 4 complete
- On the shot put photo, where her legs leave the frame, find a keypoint of the shot putter that the model still places but scores under 0.30. (not yet)
- On the handstand, find where the model puts his ankles. (not yet)
- Read the judo photo as one box, so both judoka are in the box the pose model reads. (not yet)
- The detector finds no person in the yoga statue. Get a skeleton for it anyway. (not yet)
Try this
- On the balance beam, switch to Heatmap and click through the gymnast’s keypoints in the table. Which bumps are tight, and which are spread out?
- Predict before you look: on the shot put photo, where will the model put her ankles, which are below the edge of the frame?
- On the handstand, look at his ankles and wrists. What kind of body did the model draw?
- Read the judo photo and the marathon start with Whole picture, then with One per person.
- The detector finds nobody in the yoga statue or the high jump. Try Whole picture on both.
What a score means
The score is not a probability. It is the height of the heatmap’s peak, which training pushed toward 1 where a keypoint clearly is, and it can pass 1: the gymnast’s right knee scores 1.06. What it does track is how clearly the heatmap picks out one spot. Hidden and cut-off keypoints usually come out low, so apps hide keypoints below a threshold; the lab starts at 0.30.
A top-down model answers for all 17 keypoints, whatever it can see. The shot putter’s photo stops at her thighs, yet the model places her knees on the bottom edge (0.18 and 0.27) and her ankles, which are out of the picture, on her thighs (0.12 and 0.29). In the marathon, a man in a jacket standing behind the pack has his legs hidden by the runners in front: his face scores above 0.8, his knees and ankles under 0.1.
Where top-down breaks
Everything depends on the boxes. Read the judo photo as one box and ViTPose returns a single skeleton scored 0.83 to 0.99, drawn mostly on the judoka on the left; the judoka on the right gets no skeleton, and nothing in the scores says a second person was missed. Read the marathon start as one box and every keypoint scores 0.27 or less, scattered across the pack.
The detector has its own limits. It finds 34 people in the marathon photo and in the balance beam photo, where most are spectators, so a phone would need 34 passes of the pose model; the lab reads only the 8 the detector is most sure of. It finds nobody in the yoga statue or in the high jump, where the athlete lies almost flat over the bar. Read as one box, the statue gets every keypoint at 0.77 or more, on the right parts of its body: the pose model was fine, and the detector was the step that failed. The high jumper gets scores from 0.18 to 0.63.
How the crop is made, how keypoints are refined, and how pose models are scoredOptional
The crop. ViTPose wants a 3 by 4 picture. The box is widened (or made taller) until it has that shape, then enlarged by a quarter (an eighth on each side) so arms and feet near the edge of the box are included, and that region is resampled to 192 by 256 pixels. Parts of the region outside the photo are black. This site does the crop itself, following Hugging Face Transformers’ ViTPose processor, and on a test picture matches it level for level in all but 1 of 36,864 values checked, which is one level off.
The refinement. The highest cell alone is only accurate to the size of a cell, 4 pixels of the crop, which can be 20 pixels of the photo for a large box. DARK (Feng Zhang and colleagues, 2020) does better: it blurs the heatmap slightly, takes its logarithm, which turns a Gaussian bump into a parabola, and takes one Newton step from the peak to the parabola’s top. The keypoint then maps back to the photo with the “unbiased” convention (Junjie Huang and colleagues, 2020), where the first and last cells sit exactly on the region’s edges. On the eight photos, the lab’s keypoints match Hugging Face’s PyTorch implementation, running the original 32-bit weights, to within 0.52 pixels, with scores within 0.002; the lab runs a 16-bit export.
The model. ViTPose+ small is one of the ViTPose++ models (Xu and colleagues, 2022), trained on six datasets at once: COCO, AI Challenger, and MPII for people, COCO-WholeBody for faces, hands, and feet, and AP-10K and APT-36K for animals. Each feed-forward layer has a part shared by every dataset and a part for each dataset; the lab’s export uses COCO’s. Its paper reports 75.8 average precision on COCO’s validation set with a detector’s boxes, as good as ViTPose-B, which is about four times larger.
The score. COCO compares a predicted pose with a labeled one by Object keypoint similarityCOCO's 0 to 1 score for how close a predicted pose is to a labeled one. It allows more error for larger people and for keypoints labelers place less consistently, such as hips. It plays the role IoU plays for boxes.Open in glossary, which plays the role intersection over union plays for boxes: each keypoint’s distance counts relative to the person’s size, with more slack for hips and knees, which labelers place less consistently than eyes. Average precision is then computed at ten similarity thresholds, from 0.5 to 0.95 in steps of 0.05, and averaged.
Reading a keypoint from its heatmapPython with NumPy
import numpy as np
# A pose model's heatmap for one keypoint: 64 rows by 48 columns, a quarter of
# its 256 x 192 input. Training taught it to draw a bump where the keypoint is;
# here we draw one at column 30.4, row 20.7, peaking at 0.9.
rows, cols = np.mgrid[0:64, 0:48]
heatmap = 0.9 * np.exp(-((cols - 30.4) ** 2 + (rows - 20.7) ** 2) / (2 * 2.0 ** 2))
# Take the peak: the highest cell, and its value as the confidence.
row, col = np.unravel_index(np.argmax(heatmap), heatmap.shape)
print(f"peak at column {col}, row {row}, score {heatmap[row, col]:.2f}")
# Refine to a fraction of a cell. The log of a Gaussian bump is a parabola,
# so fit one through the peak and its neighbors on each axis.
log = np.log(heatmap)
dx = (log[row, col + 1] - log[row, col - 1]) / 2
dxx = log[row, col + 1] - 2 * log[row, col] + log[row, col - 1]
dy = (log[row + 1, col] - log[row - 1, col]) / 2
dyy = log[row + 1, col] - 2 * log[row, col] + log[row - 1, col]
x, y = col - dx / dxx, row - dy / dyy
print(f"refined to column {x:.2f}, row {y:.2f}")
# Map back to the photo. The person's box became a region 300 x 400 pixels
# centered at (500, 350); the heatmap's first and last cells sit on its edges.
cx, cy, w, h = 500, 350, 300, 400
px = cx - w / 2 + x * w / (48 - 1)
py = cy - h / 2 + y * h / (64 - 1)
print(f"keypoint in the photo: x {px:.1f}, y {py:.1f}")peak at column 30, row 21, score 0.87
refined to column 30.40, row 20.70
keypoint in the photo: x 544.0, y 281.4The snippet’s refinement fits each axis separately, which is exact for a clean bump like this one. The lab’s code, like ViTPose’s, first blurs the heatmap and takes a full two-dimensional step, which copes better with the lumpy heatmaps of real photos.
Key ideas
- Pose estimation reports named keypoints, 17 per person in COCO’s scheme, not an outline of the body.
- Top-down systems box each person with a detector, then read each box with a pose model; their cost grows with the number of people, and a missed or shared box means a missed or merged pose. Bottom-up systems find all keypoints first and group them into people.
- ViTPose draws one heatmap per keypoint; the keypoint is the refined peak, and the score is the peak’s height.
- A top-down model always answers for all 17 keypoints. A low score flags hidden or cut-off keypoints, but a high score is no guarantee: on poses unlike its training data, such as a handstand, it can be confidently wrong.
Photos, all CC0 on Wikimedia Commons: Balance beam by Anefo (Nationaal Archief); Hurdles by Anefo (Nationaal Archief); Marathon start by Anefo (Nationaal Archief); Judo by Ron Kroon for Anefo (Nationaal Archief); Handstand by Nationaal Archief; Shot put by Anefo (Nationaal Archief); High jump by Eric Koch for Anefo (Nationaal Archief); Yoga statue by Apurv013. The Dutch National Archives (Nationaal Archief), which holds the rights to the Anefo photos and the handstand, released them into the public domain.