How computers see

To you, the sample photo below shows a boxer standing on a lawn. To a model it is 2,095,104 numbers: 1,024 columns by 682 rows of PixelOne tiny square of a digital image. A grayscale pixel is stored as one brightness number; a color pixel as three (red, green, and blue).Taught in Turning things into numbersOpen in glossary, each stored as three numbers. Everything a vision model “knows” about the dog has to be computed from those numbers.

This lesson looks at the picture the way two kinds of vision model do. A convolutional network looks through a small window. A vision transformer cuts the picture into patches and treats them like words.

A picture is a grid of numbers

A color pixel is three numbers: how much red, green, and blue light it holds, each usually from 0 to 255. A photo is a grid of them, height by width by 3. A 12-megapixel phone photo is 36 million numbers, far more than most models take in, so nearly every vision model starts by shrinking the picture. The classifier in the next lesson reads 224 × 224 pixels, about 150,000 numbers.

What a convolution sees

The convolutions lesson slid a 3 × 3 Kernel (convolution)The small grid of weights, often 3 x 3, that a convolution slides across an image. In a convolutional network the kernel values are learned.Taught in ConvolutionsOpen in glossary over an image. Each output pixel depends on just the nine pixels under the window. That region is the output’s Receptive fieldThe part of the input image that can affect one output of a layer. One 3 × 3 convolution sees 3 × 3 pixels; stacking layers widens it, while self-attention reaches the whole image in one layer.Open in glossary. A convolutional network widens it gradually: a second 3 × 3 layer on top of the first sees 5 × 5 pixels, a third sees 7 × 7, and shrinking steps between layers widen it faster. Only deep in the network does one output depend on most of the image.

What a vision transformer sees

In 2020, Alexey Dosovitskiy and colleagues at Google tried applying a standard TransformerA neural network architecture built from stacked layers of attention and feed-forward networks, introduced by Vaswani et al. in 2017. Most modern language models are transformers.Taught in Attention, step by stepOpen in glossary to images “with the fewest possible modifications”. Their Vision transformerA transformer that reads an image by cutting it into square patches, turning each patch into a token vector, and attending across all the patches at once. Often shortened to ViT.Open in glossary (ViT) turns a picture into a sequence in four steps:

Cut the image into square patches, for example 16 × 16 pixels.
Flatten each patch into one row of numbers: 16 × 16 pixels × 3 colors = 768.
Multiply each row by the same learned matrix to get one token vector per patch, just as a language model looks up a vector per word.
Add a learned Position encodingInformation about where each token sits in the sequence, given to a transformer because attention on its own ignores word order.Taught in Where words areOpen in glossary to each token, because attention on its own ignores order, and put one extra learned token, the classification token, in front.

From then on, it is the transformer from the transformers track. In the very first layer, Self-attentionAttention in which the queries, keys, and values all come from the same sequence, so every token can draw information from every other token in it.Taught in Attention, step by stepOpen in glossary lets every patch look at every other patch, so the receptive field is the whole image at once.

Try this

With Pixels, tap the dog’s coat, then the grass. Every color you see is three numbers, and nearby pixels have nearly the same numbers.
With Convolution and Vertical edges, tap a flat stretch of lawn, then the edge of the dog’s back leg. The output is near 0 where nothing changes and large where brightness changes from left to right.
With Patches, try each patch size and watch the token count and the attention scores. Which grows faster?
Pick the Woodblock print and look at the patches view. Then try your own photo or the camera.

Cutting an image into patch tokensPython with NumPy

import numpy as np

rng = np.random.default_rng(0)
image = rng.integers(0, 256, size=(224, 224, 3))  # height, width, color

def patchify(img, p):
    h, w, c = img.shape
    grid = img.reshape(h // p, p, w // p, p, c).transpose(0, 2, 1, 3, 4)
    return grid.reshape(-1, p * p * c)  # one row per patch, in reading order

for p in (32, 16, 8):
    tokens = patchify(image, p)
    n = tokens.shape[0]
    print(f"patch {p:2d}: {n:3d} tokens of {tokens.shape[1]:4d} numbers, "
          f"{n * n:7,d} attention scores")

# Token 1 is the second block along the top row, flattened row by row.
tokens = patchify(image, 16)
print(np.array_equal(tokens[1], image[0:16, 16:32].reshape(-1)))

# The patch embedding: one learned matrix turns each 768-number row
# into a 192-number token vector (DeiT-Tiny's width).
E = rng.normal(0, 0.02, size=(768, 192))
print((tokens @ E).shape)

# The same thing as a convolution whose 16 x 16 kernel jumps 16 pixels:
# each of the 192 kernels is one column of E, reshaped to 16 x 16 x 3.
kernel0 = E[:, 0].reshape(16, 16, 3)
print(np.isclose(np.sum(image[0:16, 16:32] * kernel0), (tokens @ E)[1, 0]))

patch 32:  49 tokens of 3072 numbers,   2,401 attention scores
patch 16: 196 tokens of  768 numbers,  38,416 attention scores
patch  8: 784 tokens of  192 numbers, 614,656 attention scores
True
(196, 192)
True

Tokens cost quadratically

A 224-pixel square in patches of side $p$ gives $(224/p)^2$ tokens. Attention scores every token against every token, so one head in one layer computes the square of that. At 16 pixels that is 196 tokens and 38,416 scores; at 8 pixels, 784 tokens and 614,656 scores. Smaller patches keep finer detail, but the cost grows 16 times each time the patch side halves. That is why vision transformers use patches of 14 to 32 pixels rather than single pixels.

The patch embedding

For an image of $H \times W$ pixels with $C$ color channels cut into $P \times P$ patches, there are $N = HW/P^2$ patches. Each flattened patch $\mathbf{x}_p^i \in \mathbb{R}^{P^2 C}$ is multiplied by one learned matrix $\mathbf{E} \in \mathbb{R}^{(P^2 C) \times D}$ , and the sequence the transformer reads is

\mathbf{z}_0 = [\,\mathbf{x}_\text{class};\ \mathbf{x}_p^1 \mathbf{E};\ \mathbf{x}_p^2 \mathbf{E};\ \dots;\ \mathbf{x}_p^N \mathbf{E}\,] + \mathbf{E}_\text{pos}.

Multiplying a flattened patch by one column of $\mathbf{E}$ is the same as laying a $P \times P \times C$ kernel over that patch and summing the products. So the patch embedding is a convolution whose kernel size and stride both equal $P$ , and that is how most code implements it. For the classifier in the next lesson, DeiT-Tiny, $P = 16$ , $C = 3$ , and $D = 192$ .

Why both kinds of model exist

A convolution builds in two assumptions: nearby pixels matter most, and a pattern means the same thing wherever it appears. Attention builds in neither, so a vision transformer must learn them from data. The ViT authors found that trained on ImageNet’s 1.3 million labeled images, their models came out “a few percentage points below ResNets of comparable size”, but pretrained on 14 million or 300 million images they approached or beat the best convolutional networks. Two months later, Hugo Touvron and colleagues at Facebook AI (now Meta) trained competitive vision transformers on ImageNet alone, with heavy data augmentation and a distillation token that learns from a convolutional teacher. That model family, DeiT, is the classifier you will use next.

Today both designs are in wide use, and many models mix them. The detector in Recognizing objects with your camera and the depth model in Depth from a single image are built on vision transformers, and the patch embedding itself is a convolution.

The picture a model actually reads

Before any of this, each model resizes the picture its own way, and that changes what it can see.

The classifier in the camera lesson (DeiT) stretches every picture to exactly 224 × 224, so a tall print gets squeezed.
CLIP, in Images and words together, scales the short side to 224 and keeps the middle square, so whatever sits near the long edges is cut off.
The detector and the depth model keep the shape: they scale the short side to a fixed length (512 and 518 pixels), so a wider picture gets more patches.

The lab’s patches view stretches the picture like the classifier, using your browser’s highest-quality resizing. The model in your browser resizes at a lower quality setting, and it was trained on pictures resized with bicubic resampling, so the exact numbers it reads differ slightly from the ones shown.

Key ideas

A color image is a height × width × 3 grid of numbers, and every model shrinks it to a fixed size first.
One convolution output sees only a small window; a convolutional network widens its view layer by layer.
A vision transformer cuts the image into patches, turns each into a token vector with one learned matrix, adds positions, and lets every patch attend to every other from the first layer.
Smaller patches keep more detail but cost quadratically more attention.
Each model resizes, stretches, or crops in its own way, and that decides what it can see.

How computers see

In this lesson you will