Latent

Find a body's keypoints

A person detector boxes each person, then ViTPose+ finds 17 keypoints in each box, from the nose to the ankles. See the heatmap behind every keypoint, and where pose estimation fails.

IntermediateExplained in Finding a body's pose

Find a body's keypoints

A detector boxes each person, then a pose model finds 17 keypoints inside each box.

Boxes the pose model reads
Show
0.30
Photos

Your photos and camera frames are processed on this device and never uploaded.

left side right side nose The person's own left and right. Tap a box to choose a person.

1. Find each person YOLOS-Tiny

Recorded from YOLOS-Tiny

Loading the model's recorded outputs for the eight bundled photos.

About the download

Box every person in your own photo or the camera, so the pose model can read each one. This uses YOLOS-Tiny, a 6.5-million-parameter vision transformer (2021) that finds and boxes the 80 kinds of everyday objects in the COCO dataset.

The first time, your browser downloads about 27 MB of model files from Hugging Face, plus a 14 MB runtime from this site. Both are cached for later visits. Everything runs on your device; no photo or camera frame is sent anywhere.

Manage downloads

Loading the recorded results

2. Find 17 keypoints in each box ViTPose+ small

Recorded from ViTPose+ small

Loading the model's recorded outputs for the eight bundled photos.

About the download

Find the body keypoints of each person in your own photo or the camera. This uses ViTPose+ small, a vision transformer from the University of Sydney (2022) that finds 17 body keypoints, from the nose to the ankles, for one person in a box. About 24 million of its 33 million parameters compute COCO's 17 body keypoints; the rest are the parts of each layer trained for its other datasets, such as faces and hands or animals.

The first time, your browser downloads about 67 MB of model files from Hugging Face, plus a 14 MB runtime from this site. Both are cached for later visits. Everything runs on your device; no photo or camera frame is sent anywhere.

Manage downloads

Loading the recorded results

Boxes readn/a
Keypoints shownn/a
Time per boxn/a

Challenges

0 of 4 complete

  1. On the shot put photo, where her legs leave the frame, find a keypoint of the shot putter that the model still places but scores under 0.30. (not yet)
  2. On the handstand, find where the model puts his ankles. (not yet)
  3. Read the judo photo as one box, so both judoka are in the box the pose model reads. (not yet)
  4. The detector finds no person in the yoga statue. Get a skeleton for it anyway. (not yet)

Try this

  • On the balance beam, raise the threshold slowly and watch which keypoints disappear first.
  • Switch to Heatmap and step through the keypoints in the table. Where is the bump tight, and where is it spread out?
  • On the shot put photo, find the keypoints below the edge of the frame. Where does the model put them, and how sure is it?
  • Read the marathon start with Whole picture, then with One per person.
  • With both models downloaded, stand in front of the camera and turn sideways, cross your arms, or hide a hand behind your back.
Reading a keypoint from its heatmapPython with NumPy
import numpy as np

# A pose model's heatmap for one keypoint: 64 rows by 48 columns, a quarter of
# its 256 x 192 input. Training taught it to draw a bump where the keypoint is;
# here we draw one at column 30.4, row 20.7, peaking at 0.9.
rows, cols = np.mgrid[0:64, 0:48]
heatmap = 0.9 * np.exp(-((cols - 30.4) ** 2 + (rows - 20.7) ** 2) / (2 * 2.0 ** 2))

# Take the peak: the highest cell, and its value as the confidence.
row, col = np.unravel_index(np.argmax(heatmap), heatmap.shape)
print(f"peak at column {col}, row {row}, score {heatmap[row, col]:.2f}")

# Refine to a fraction of a cell. The log of a Gaussian bump is a parabola,
# so fit one through the peak and its neighbors on each axis.
log = np.log(heatmap)
dx = (log[row, col + 1] - log[row, col - 1]) / 2
dxx = log[row, col + 1] - 2 * log[row, col] + log[row, col - 1]
dy = (log[row + 1, col] - log[row - 1, col]) / 2
dyy = log[row + 1, col] - 2 * log[row, col] + log[row - 1, col]
x, y = col - dx / dxx, row - dy / dyy
print(f"refined to column {x:.2f}, row {y:.2f}")

# Map back to the photo. The person's box became a region 300 x 400 pixels
# centered at (500, 350); the heatmap's first and last cells sit on its edges.
cx, cy, w, h = 500, 350, 300, 400
px = cx - w / 2 + x * w / (48 - 1)
py = cy - h / 2 + y * h / (64 - 1)
print(f"keypoint in the photo: x {px:.1f}, y {py:.1f}")
peak at column 30, row 21, score 0.87
refined to column 30.40, row 20.70
keypoint in the photo: x 544.0, y 281.4

Photos, all CC0 on Wikimedia Commons: Balance beam by Anefo (Nationaal Archief); Hurdles by Anefo (Nationaal Archief); Marathon start by Anefo (Nationaal Archief); Judo by Ron Kroon for Anefo (Nationaal Archief); Handstand by Nationaal Archief; Shot put by Anefo (Nationaal Archief); High jump by Eric Koch for Anefo (Nationaal Archief); Yoga statue by Apurv013. The Dutch National Archives (Nationaal Archief), which holds the rights to the Anefo photos and the handstand, released them into the public domain.

Try "embedding", "softmax", "overfitting", or "backpropagation".