From pixels to patches
Pick a picture and see it the way a vision model does: a grid of numbers, a convolution's small window, and a vision transformer's patches.
Your photos and camera frames are processed on this device and never uploaded.
Pictureloading
Tokens196 patches of 768 numbers
Attention scores per head, per layer38,416
Squeezed to a square1.00 to 1
Challenges
0 of 3 complete
- Cut the 224 × 224 input into exactly 49 tokens. (not yet)
- Choose a patch size that makes one attention head compute more than 100,000 scores per layer. (not yet)
- Find a picture that the classifier would squeeze by more than 2 to 1 to make it square. (not yet)
Try this
- With Pixels, tap different parts of the picture and read the red, green, and blue values.
- With Convolution, compare Vertical edges and Horizontal edges on the corridor photo.
- With Patches, step through the patch sizes and watch how fast the attention scores grow.
- Use your own photo or the camera. Nothing leaves your device.
Cutting an image into patch tokensPython with NumPy
import numpy as np
rng = np.random.default_rng(0)
image = rng.integers(0, 256, size=(224, 224, 3)) # height, width, color
def patchify(img, p):
h, w, c = img.shape
grid = img.reshape(h // p, p, w // p, p, c).transpose(0, 2, 1, 3, 4)
return grid.reshape(-1, p * p * c) # one row per patch, in reading order
for p in (32, 16, 8):
tokens = patchify(image, p)
n = tokens.shape[0]
print(f"patch {p:2d}: {n:3d} tokens of {tokens.shape[1]:4d} numbers, "
f"{n * n:7,d} attention scores")
# The patch embedding: one learned matrix turns each 768-number row
# into a 192-number token vector (DeiT-Tiny's width).
tokens = patchify(image, 16)
E = rng.normal(0, 0.02, size=(768, 192))
print((tokens @ E).shape)patch 32: 49 tokens of 3072 numbers, 2,401 attention scores
patch 16: 196 tokens of 768 numbers, 38,416 attention scores
patch 8: 784 tokens of 192 numbers, 614,656 attention scores
(196, 192)Sample photos, all CC0 on Wikimedia Commons: Dog by Joselodos; Cat by Vyshnavi medicharla; Cat on a motorcycle by CAPTAIN RAJU; Breakfast by Andy Li; Street by Mucklagh; Cafe by Andy Li; Corridor by Sindugab; Woodblock print by Utagawa Hiroshige (Metropolitan Museum of Art).