Where the classifier looks

In the recognition lesson, the classifier called a temple corridor a prison at 78% and a motorcycle a moped. You cannot ask it why. It has no reasons to report, only 1,000 numbers that come out of 12 layers.

You can run an experiment instead. Hide part of the picture, run the model again, and see what changes. Do that for every part, and you get a map of which regions the answer depends on. Matthew Zeiler and Rob Fergus used this in 2014 to check that an image classifier was finding the object rather than guessing from the background.

Covering one square at a time

Lay a grid over the picture. For each square:

Replace that square with something that carries no detail, such as plain gray.
Run the classifier on the changed picture.
Record how much the label’s probability fell or rose.

A 7 by 7 grid takes 49 runs of the model, plus one on the untouched picture. Color each square by its change and you have an Occlusion mapA map of how much a model's output changes when each region of its input is covered in turn, made by running the model once per covered region.Open in glossary: red where covering the square lowers the label, blue where it raises it.

The classifier is DeiT-Tiny again. It reads a picture stretched to 224 × 224 pixels as 196 patches of 16 × 16, so the smallest squares in the lab cover exactly one patch. The gray is ImageNet’s average color, which is what zero means after the model’s input normalization. The lab runs every covered copy on your device.

Try this

On the dog, find the square that hurts “boxer” most. Before you switch to one-patch squares, predict whether one patch can do as much damage.
On the street or the cafe, cover squares until the top label changes. Then try the same on the dog.
Try the corridor with the largest squares. Look for any square that brings “prison” down by much.
Pick a square, then switch the fill between gray and blur. Does the classifier care what replaced it?
Choose a different label to explain, such as the dog’s second label. Its map can point somewhere else.

What the maps show

On the dog at 7 by 7, the square over the head is the one that matters most: covering it takes “boxer” from 77% to 32%. Most squares over the lawn move it by less than 10 points. For this photo, the answer rests mostly where a person would look.

That is not always so. The moped label on the motorcycle photo leans hardest on the bottom left corner of the 4 by 4 grid, the rear wheel and the wire guard over it, which costs 32 points when covered. Covering the cat does not hurt it at all; one of the cat’s squares even raises “moped” by 7 points. An occlusion map can confirm a sensible answer, and it can also show a label resting on a detail you would not have guessed.

Labels that won narrowly are fragile. The street’s “parking meter” (24%) and the cafe’s “barbershop” (32%) each lose the top spot when one large square is covered: the cafe becomes a library. The dog’s “boxer” survives every square at every size.

Clever Hans

In the early 1900s a horse in Berlin named Hans appeared to do arithmetic, tapping out answers with a hoof. In a study published in 1907, Oskar Pfungst showed that Hans was reading tiny, unintended cues from the people asking: when Hans could not see the questioner, his answers were almost never right. The answers had been right for the wrong reason.

Models do this too. Sebastian Lapuschkin and colleagues (2019) inspected a classifier trained on PASCAL VOC 2007, a standard photo benchmark, with a relevance map, a cousin of the occlusion map. Its decisions for “horse” focused on the lower left corner, where about one fifth of the horse photos carried a photo agency’s source tag. Removing the tag from a horse photo removed the horse. Adding the tag to a picture of a car turned the car into a horse.

This is Shortcut learningWhen a model learns a rule that scores well on its test data because of a quirk of that data, such as a watermark on one class, and fails once the quirk is gone.Open in glossary: a rule that scores well on the benchmark because of a quirk of its data, and fails as soon as the quirk is gone. Accuracy on more photos from the same collection cannot catch it, because those photos share the quirk. Covering, removing, and adding the suspected cue can.

What an occlusion map misses

Every choice in the experiment shapes the map.

Square size. At one patch per square, no single square on the dog takes more than 19 points from “boxer”, because the rest of the head is still visible. Large squares find regions; small squares find what no neighbor can replace.
The fill. A gray square is unlike anything in the training photos, so part of any change can come from the square itself. Blurring keeps the colors and removes the detail, and it can disagree with gray. On the woodblock print, blurring one square raises “envelope” by 39 points.
Evidence that is spread out. The corridor’s “prison” (78%) and the cat’s “tabby” (64%) lose less than a quarter of their probability to any one large square. Their evidence, rows of pillars and fur everywhere, is repeated across the picture, and covering one copy leaves the others. The map looks flat while the label depends on the whole pattern. The code below builds a tiny classifier where this happens exactly.
Cost. One run per square: 196 runs for one-patch squares. That is easy for a small classifier and slow for a large model.

An occlusion map, and evidence it missesPython with NumPy

import numpy as np

def softmax(z):
    e = np.exp(z - z.max())
    return e / e.sum()

# A toy classifier for 4 x 4 pictures with two labels. "Stripe" scores the
# brightest row; "blob" scores the middle four pixels. Pixels are already
# normalized, so 0 is the average color, the gray a covered square gets.
def probs(img):
    stripe = 3 * img.sum(axis=1).max() / 4
    blob = 3 * img[1:3, 1:3].mean()
    return softmax(np.array([stripe, blob]))

def occlusion_map(img, label, size=2):
    base = probs(img)[label]
    n = img.shape[0] // size
    drops = np.zeros((n, n))
    for r in range(n):
        for c in range(n):
            covered = img.copy()
            covered[r*size:(r+1)*size, c*size:(c+1)*size] = 0  # gray square
            drops[r, c] = base - probs(covered)[label]
    return base, drops

img = np.zeros((4, 4))
img[0, :] = 1.0            # a bright stripe along the top
base, drops = occlusion_map(img, label=0)
print(f"stripe with nothing covered: {base:.2f}")
print("drop when each 2 x 2 square is covered:")
print(np.round(drops, 2))

# The same evidence twice: a second stripe along the bottom. The classifier
# takes the brightest row, so covering either stripe leaves the other.
img[3, :] = 1.0
base, drops = occlusion_map(img, label=0)
print(f"two stripes, nothing covered: {base:.2f}")
print(np.round(drops, 2))

stripe with nothing covered: 0.95
drop when each 2 x 2 square is covered:
[[0.13 0.13]
 [0.   0.  ]]
two stripes, nothing covered: 0.95
[[0. 0.]
 [0. 0.]]

With two stripes, every square’s drop is zero, yet the label depends entirely on the stripes. Cover both and it falls. Single-square occlusion only sees what one region contributes on its own.

Other ways to map what a model relies onOptional

Occlusion needs only the model’s outputs, which makes it easy to trust and slow to run. Other methods use the network’s internals.

A Saliency mapA heatmap over a model's input showing which parts most affect one output, usually computed from gradients of that output with respect to the input.Open in glossary in the original sense (Simonyan, Vedaldi, and Zisserman, 2013) takes the gradient of a label’s score with respect to each input pixel: how much a tiny change to that pixel would move the score. It needs one backward pass instead of hundreds of forward passes, but tiny changes are not the same as removing a region. Grad-CAM (Selvaraju and colleagues, 2017) weights a convolutional network’s last feature maps by their gradients to get a coarse map per label.

A good-looking map is not proof. Julius Adebayo and colleagues (2018) found that some popular gradient-based maps barely changed when the network’s trained weights were replaced with random ones, so those maps were mostly showing edges in the picture rather than anything the model had learned. Intervening, as occlusion does, and checking that the output really changes is the safer habit.

The same habit runs through the interpretability track, where you switch off parts of a language model and measure what breaks.

Key ideas

An occlusion map covers one region at a time, runs the model again, and records how much a label falls or rises.
A big drop means the answer depended on that region for this picture. It does not mean the model ignored everything else.
Narrow winners flip when one region is covered; confident labels whose evidence is spread out survive any single square.
The square size, the fill, and repeated evidence all change the map, so read it as the result of an experiment, not as the model’s gaze.
Covering, removing, and adding a suspected cue is how shortcut learning, such as a classifier that spots horses by a source tag, gets caught.

Sample photos, all CC0 on Wikimedia Commons: Dog by Joselodos; Cat by Vyshnavi medicharla; Cat on a motorcycle by CAPTAIN RAJU; Breakfast by Andy Li; Street by Mucklagh; Cafe by Andy Li; Corridor by Sindugab; Woodblock print by Utagawa Hiroshige (Metropolitan Museum of Art).

Where the classifier looks

In this lesson you will

Covering one square at a time