Depth from a single image

Close one eye and look around. Without the second eye’s slightly different view, you still know the cup is nearer than the wall. You are using cues in the flat image: nearer things block farther ones, parallel lines such as the edges of a floor converge toward the horizon, familiar objects look smaller when far away, textures get finer with distance, and distant hills look hazy.

A neural network can learn the same cues from data and turn a single photo into a depth map, a guess at how near every pixel is. This is Monocular depth estimationGuessing how far away every pixel is from a single photo, using learned cues such as overlap, perspective, and familiar sizes. Most models give relative depth, an order with no units, rather than meters.Open in glossary, and it runs in your browser below.

One picture is ambiguous

A camera flattens a three-dimensional scene onto a sensor. Scale the whole scene up and move it back, and the picture does not change: a toy car close to the lens and a real car down the street can look the same. So a single photo cannot fix distances in meters. A model that reports meters must assume things the pixels do not say, such as how big cars usually are or what lens took the photo.

Depth Anything V2, the model in the lab, does not try. It predicts relative inverse depth: one over the distance, up to a scale and an offset it never states. For each picture, the true inverse depth is roughly the model’s output times some unknown $s$ plus some unknown $t$ . Larger outputs mean nearer, and the order of any two points can be read off. How much nearer, or how far in meters, cannot.

How the model learned

Depth Anything V2 (Lihe Yang and colleagues, 2024) learned in two stages.

A large teacher model was trained only on 595,000 synthetic images from five rendered datasets, where the renderer knows the exact depth of every pixel. Depth labels for real photos, from laser scanners or stereo cameras, are noisy, coarse, and missing on glass and sky.
The teacher then labeled 62 million real photos, and smaller student models learned from those labels. Learning from a teacher’s outputs is Knowledge distillationTraining a smaller student model to reproduce a larger teacher model's outputs, so the student learns from the teacher's predictions instead of, or as well as, human labels.Open in glossary. The student in the lab, the smallest, has 25 million parameters.

Each student is a vision transformer with 14-pixel patches (DINOv2) followed by a decoder (DPT) that turns token features from several layers back into a full-resolution map. Training compares prediction and label only after lining them up with the best scale and offset, the scale- and shift-invariant loss that René Ranftl and colleagues proposed in MiDaS (2019) to mix depth datasets, so the model is never asked for the units the photos cannot give.

Try this

On the corridor, put A on the floor near the bottom and B at the far end. Then move B along a row of pillars.
Pick the street and tap the sky. Where is the far end of the model’s scale?
Pick the woodblock print. The paper is flat; what does the model say?
Switch to 3D relief and turn it. Which parts stick out, and which look flattened?
With the camera, try a mirror or a window, and a photo of a photo on a screen.

Relative depth, its order, and its alignmentPython with NumPy

import numpy as np

# Five points at known distances, in meters.
depth = np.array([1.0, 2.0, 4.0, 8.0, 50.0])
inverse = 1 / depth  # what the model is trained to predict, up to scale and offset

# A relative-depth model's output: inverse depth times an unknown scale,
# plus an unknown offset, plus a little error.
rng = np.random.default_rng(0)
output = 3.0 * inverse + 0.4 + rng.normal(0, 0.01, depth.size)
print("model output:", np.round(output, 3))
print("nearest first, true :", np.argsort(depth))
print("nearest first, model:", np.argsort(-output))

# Benchmarks align each prediction to the truth with the best scale and
# offset (least squares) before measuring error.
A = np.stack([output, np.ones_like(output)], axis=1)
scale, shift = np.linalg.lstsq(A, inverse, rcond=None)[0]
print(f"fitted scale {scale:.3f}, offset {shift:.3f}")
print("recovered distances:", np.round(1 / (scale * output + shift), 1))

model output: [3.401 1.899 1.156 0.776 0.455]
nearest first, true : [0 1 2 3 4]
nearest first, model: [0 1 2 3 4]
fitted scale 0.333, offset -0.133
recovered distances: [ 1.   2.   4.   8.  54.4]

Reading the map

The lab shows each pixel’s place between the farthest point in that picture (0) and the nearest (1). That is all the model commits to, so it is all the lab shows. Two pictures’ numbers cannot be compared, and within one picture only the order is solid.

The far end is usually the sky or a vanishing point. Inverse depth turns infinite distance into a finite number, zero, which is one reason models predict it rather than depth itself. The cost is that far distances are fragile: in inverse depth, 50 meters and 100 meters differ by only 0.01, so a tiny error in the output becomes a large error in meters. In the code above, an error of about 0.01 barely moves the near points but turns 50 meters into 54.4.

The 3D relief raises each part of the picture by its relative inverse depth. It is a relief, not a reconstruction: real geometry would need the scale and offset, and the focal length of the camera.

Where it fails

Pictures of depth. The woodblock print is flat paper, but its bird and branch overlap the background, and the model reports them as nearer. A photo of a photo, a poster, or a painted mural fools it in the same way. It infers depth from cues; it does not measure.
Mirrors and glass. A mirror shows a whole room inside a flat surface. Reflections and transparent objects are classic failures for depth models; the Depth Anything V2 authors report clear improvements over version 1, so try one with the camera and see.
Scenes unlike its training data. The paper names a gap between synthetic and real images, which the pseudo-labeled real photos are meant to close. Unusual viewpoints, such as the breakfast table seen from straight above, are worth testing.

Key ideas

A single photo cannot fix distances in meters; any absolute answer comes from assumptions about sizes and cameras.
Depth Anything V2 predicts relative inverse depth: the order of points is meaningful, ratios and units are not.
Inverse depth puts the sky near zero and makes far distances fragile.
The model learned from 595,000 synthetic images through a teacher that labeled 62 million real photos.
It infers depth from cues, so flat pictures of depth, mirrors, and unusual scenes fool it.

Sample photos, all CC0 on Wikimedia Commons: Dog by Joselodos; Cat by Vyshnavi medicharla; Cat on a motorcycle by CAPTAIN RAJU; Breakfast by Andy Li; Street by Mucklagh; Cafe by Andy Li; Corridor by Sindugab; Woodblock print by Utagawa Hiroshige (Metropolitan Museum of Art).