Diffusion on images

In the previous lesson the data were dots in two dimensions, and the reverse process could be computed exactly because the noisy data had a known formula. Images have no such formula. A real diffusion modelDiffusion modelA generative model that learns to reverse a gradual noising process, so it can turn pure random noise into new data such as images, step by step.Taught in Diffusion: from noise to dataOpen in glossary has to learn the reverse process from examples. This lesson runs one: a small network trained on 60,000 handwritten digits, generating new digits in your browser, step by step.

The same forward process, in 784 dimensions

A 28 × 28 grayscale digit is 784 numbers. Scale each pixel to the range −1 (blank background) to 1 (full ink), and the forward process is the one you already know:

Every pixel gets its own independent noise. The noise scheduleNoise scheduleHow much noise a diffusion model's forward process adds at each step. DDPM's schedule raises the per-step noise variance beta from 0.0001 to 0.02 over 1,000 steps, which fixes how much of the image survives at every step.Open in glossary is DDPM’s (Ho et al., 2020): 1,000 steps, with the noise variance added at step $t$, $\beta_t$, rising in a straight line from 0.0001 to 0.02, and $\bar\alpha_t = \prod_{s=1}^{t} (1 - \beta_s)$. It is the previous lesson’s schedule in its original, discrete form: there $\beta(t)$ rose from 0.1 to 20 as time ran from 0 to 1, and splitting that time into 1,000 steps gives each step one thousandth of it.

import numpy as np

# DDPM's linear schedule: beta rises from 0.0001 to 0.02 over 1000 steps.
betas = np.linspace(1e-4, 0.02, 1000)
abar = np.cumprod(1 - betas)  # share of the signal's variance left after step t


def noisy(x0, eps, t):  # the forward process, t from 1 to 1000
    a = abar[t - 1]
    return np.sqrt(a) * x0 + np.sqrt(1 - a) * eps


for t in (1, 260, 500, 1000):
    a = abar[t - 1]
    print(f"t = {t:4d}: signal kept {np.sqrt(a):.3f}, noise {np.sqrt(1 - a):.3f}")

# A 4-pixel "image" on the model's [-1, 1] scale, and the noise mixed into it.
x0 = np.array([-1.0, -1.0, 0.6, 1.0])
eps = np.array([0.5, -1.2, 0.3, 2.0])
xt = noisy(x0, eps, 500)
print("x_500:", np.round(xt, 3))

# If you knew the noise exactly, you could undo the mixing: this is the
# "clean estimate" the lab shows, computed from the network's noise guess.
a = abar[499]
print("clean estimate:", np.round((xt - np.sqrt(1 - a) * eps) / np.sqrt(a), 3))

t =    1: signal kept 1.000, noise 0.010
t =  260: signal kept 0.705, noise 0.709
t =  500: signal kept 0.280, noise 0.960
t = 1000: signal kept 0.006, noise 1.000
x_500: [ 0.2   -1.432  0.456  2.2  ]
clean estimate: [-1.  -1.   0.6  1. ]

The digit fades smoothly, but not evenly. By step 260 the noise is as strong as the digit, and from step 750 on less than 6% of the signal is left. Nichol and Dhariwal (2021) noticed the same thing on small images: the last quarter of this schedule is almost pure noise. It is kept here because it is the schedule DDPM and DDIM were published with, so what you see matches those papers.

What the network learns

The network $\varepsilon_\theta(x_t, t, c)$ gets the noisy image $x_t$, the step $t$, and optionally the class $c$, and is trained to guess the noise that was mixed in (Ho et al., 2020, algorithm 1):

Training needs nothing but clean digits. Pick one, pick a step at random, draw noise, form $x_t$ with the forward formula, and nudge the network toward predicting that noise. Given its noise estimate $\hat\varepsilon$, the forward formula can be solved for the clean image:

That is the clean estimate in the lab. It explains what you saw at high noise. A network trained with squared error learns the average answer for each input, so $\hat x_0$ is the average of all the clean digits that could be hiding under this noise. Around step 700 the noisy image no longer says which digit it was, so that average is a blur that looks like no digit in particular. Told that the digit is a 7, it averages over 7s only and keeps a 7’s shape. As the noise shrinks, fewer digits fit, and the average sharpens into one of them. Near step 1000 the estimate also turns grainy: dividing by $\sqrt{\bar\alpha_t} \approx 0.006$ multiplies any error in the noise estimate by more than 150.

This is the score from the previous lesson in another form: the best noise estimate is $-\sqrt{1 - \bar\alpha_t}$ times the score of the noisy data, so learning to predict noise is learning the arrows you saw there.

Running it backward

To generate, start from pure noise and repeat: ask the network for its noise estimate, compute the clean estimate, and take a step from the current image toward it. Song, Meng, and Ermon (2020, equation 12) wrote one step from $t$ to an earlier step $s$ as

where $z$ is fresh noise drawn from $\mathcal{N}(0, I)$. Put the clean estimate back at the right strength for step $s$, put back the predicted noise at the right strength, and possibly add a little fresh noise. How much fresh noise is a choice, set by a knob $\eta$ through $\sigma$:

• DDPM ($\eta = 1$, Ho et al., 2020): every step adds fresh noise. It was designed to run all 1,000 steps.
• DDIMDDIMDenoising diffusion implicit models: a way to sample from a trained diffusion model without adding fresh noise at each step, so the same starting noise always gives the same image and far fewer, larger steps work.Open in glossary ($\eta = 0$, Song et al., 2020): no fresh noise at all. Each step is deterministic, so the same starting noise always gives the same image, and the steps can be large: 50, 20, or 10 instead of 1,000.

The network is the same in both; only the sampler changes. The lab clips the clean estimate to the valid pixel range before each step, as the DDPM reference code did.

import numpy as np

betas = np.linspace(1e-4, 0.02, 1000)
abar = np.cumprod(1 - betas)


def step(x, eps_hat, t, t_prev, eta, z):
    """One reverse step from t to t_prev (Song et al. 2020, eq. 12).
    t_prev = 0 means the clean image. eta = 0 is DDIM, eta = 1 is DDPM."""
    a = abar[t - 1]
    a_prev = abar[t_prev - 1] if t_prev > 0 else 1.0
    x0_hat = np.clip((x - np.sqrt(1 - a) * eps_hat) / np.sqrt(a), -1, 1)
    eps_hat = (x - np.sqrt(a) * x0_hat) / np.sqrt(1 - a)  # agree with the clipped x0
    sigma = eta * np.sqrt((1 - a_prev) / (1 - a) * (1 - a / a_prev))
    return np.sqrt(a_prev) * x0_hat + np.sqrt(1 - a_prev - sigma**2) * eps_hat + sigma * z


# Classifier-free guidance: the network runs twice and the answers are combined.
eps_cond = np.array([0.4, -0.9, 0.1, 1.6])  # told "this is a 3"
eps_uncond = np.array([0.6, -0.7, 0.2, 1.2])  # told nothing
for scale in (0, 1, 3):
    print(f"guidance {scale}:", np.round(eps_uncond + scale * (eps_cond - eps_uncond), 2))

x = np.array([0.2, -1.4, 0.5, 2.2])  # a 4-pixel image at step t = 500
eps_hat = eps_uncond + 3 * (eps_cond - eps_uncond)
z = np.array([0.3, 0.3, -0.8, 0.1])  # fresh noise, used only by DDPM
print("DDIM, 500 to 400:", np.round(step(x, eps_hat, 500, 400, 0.0, z), 3))
print("DDPM, 500 to 499:", np.round(step(x, eps_hat, 500, 499, 1.0, z), 3))

guidance 0: [ 0.6 -0.7  0.2  1.2]
guidance 1: [ 0.4 -0.9  0.1  1.6]
guidance 3: [ 0.  -1.3 -0.1  2.4]
DDIM, 500 to 400: [ 0.315 -1.406  0.647  1.99 ]
DDPM, 500 to 499: [ 0.231 -1.363  0.42   2.196]

What you saw:

• Coarse to fine. The clean estimate is a blob, then a recognizable digit, then a digit with its final strokes. The overall shape is decided early, at high noise, and the details at the end, as in the previous lesson.
• Few large steps work with DDIM. From one starting noise, DDIM’s 10-step and 1,000-step digits differ by 0.014 per pixel on average (pixels run from 0 to 1), and its 100-step and 1,000-step digits by 0.002: the same digit. Measured over 1,000 digits, DDIM is closer to real handwriting than DDPM at every step count compared (10 to 250), and 1,000 DDPM steps land about where 20 DDIM steps do (see the table below). The DDIM paper found the same on CIFAR-10: its samples at 20 to 100 steps were comparable to 1,000-step DDPM, and the starting noise alone largely fixed the image’s high-level features.
• DDPM is random all the way down. Two 1,000-step DDPM runs from the same starting noise differ by 0.127 per pixel on average, about as much as either differs from DDIM’s digit. The fresh noise added at every step, not the starting noise, decides most of the drawing.
• The class decides the digit; the noise shapes the rest. Digits of different classes made from one starting noise share a look. Over 200 starting noises, the amount of ink in a digit (thicker or larger strokes mean more) correlated at 0.61 between classes generated from the same noise, and at 0.00 when the noises were shuffled.

Classifier-free guidance

During training, the class was replaced by a “no class” token 10% of the time, so one network learned both the conditional model (“a 7”) and the unconditional one (“any digit”). Classifier-free guidanceClassifier-free guidanceA sampling technique for conditional generative models, diffusion or token by token, that compares the model's prediction with and without the prompt and pushes further toward the prompt, trading variety for closer matching.Open in glossary (Ho and Salimans, 2022) uses both at every step:

Scale $s = 0$ ignores the class, $s = 1$ is the plain conditional model, and $s > 1$ extrapolates: it pushes each step further in the direction that separates “a 7” from “any digit”. The paper writes the same rule with a weight $w = s - 1$. Each step now costs two network calls, which the lab’s call counter shows.

In this model (DDIM, 50 steps), guidance 1 gives a digit of the requested class 95.1% of the time, guidance 2 gives 99.9%, and guidance 3 or more gives 100%, as judged by a separate classifier. Over the same range the digits drift away from real handwriting: the distance to real digits (from the table above) grows from 11.0 at guidance 1 to 49.4 at 3 and 113.8 at 8, and digits of one class become more alike. Ho and Salimans report the same trade on ImageNet: more guidance, images that match their class better but are less varied. For comparison, the Stable Diffusion pipeline in the diffusers library defaults to a guidance scale of 7.5 and 50 steps.

From digits to text-to-image

The ideas in the lab are the ones real image generators use. The scale is not. For comparison, Stable Diffusion v1 (CompVis model card and repository), and Qwen-Image, an open model from 2025 described in the next section (Qwen-Image technical report and model card):

Two differences deserve a closer look:

• Latent diffusion. Running a U-Net on 786,432 pixel values per step is expensive. Latent diffusion (Rombach et al., 2022) first trains an autoencoder, then runs diffusion on the compressed latents and decodes the result once at the end. Stable Diffusion’s autoencoder shrinks each image 8 times in each direction. The autoencoders lesson shows the same encode, work in a latent space, decode idea on digits.
• Text conditioning. Instead of a class vector, the denoiser reads the text encoder’s output for every token of the prompt (not CLIP’s single shared embedding) through cross-attention layers. Guidance then pushes toward “this prompt” and away from “no prompt”. The text encoder is CLIP’s, which the images and text lesson and Images and words together cover.

Text to image in 2025 and 2026

Newer systems change three parts of the recipe. As of October 2026, open models whose designs are published show all three; closed ones, such as OpenAI’s and Google’s image generators, do not say this much.

• A transformer instead of the U-Net. A diffusion transformerDiffusion transformerA diffusion or flow-matching model whose denoising network is a transformer that treats small patches of the noisy image or its latent as tokens, instead of a U-Net.Open in glossary (Peebles and Xie, 2022) cuts the noisy latent into small patches and treats each patch as a token, as a vision transformer does with a photo. Stable Diffusion 3 (Esser et al., 2024) put the prompt’s tokens and the picture’s patches in one sequence, with “separate weights for the two modalities” but attention across both, in place of the U-Net’s cross-attention.
• A language model reads the prompt. Stable Diffusion v1 read prompts with CLIP’s text encoder alone. Stable Diffusion 3 added T5-XXL, a 4.7-billion-parameter language model, beside two CLIP encoders. Qwen-Image reads them with Qwen2.5-VL, a 7-billion-parameter vision-language model, and FLUX.2 couples “the Mistral-3 24B parameter vision-language model with a rectified flow transformer” (Black Forest Labs, November 2025).
• A velocity instead of the noise. The network is trained to predict the direction from noise to picture along a straight path, which is the flow matching the previous lesson ends with. The clean estimate and the sampling loop work as in this lab.

Qwen-Image, released with open weights by Alibaba’s Qwen team in August 2025, shows all three (see the table above). Its denoiser is about 40,000 times the size of this lab’s, and it still runs classifier-free guidance, two network calls a step. Its successor, Qwen-Image-2.0 (May 2026), reads prompts with Qwen3-VL, uses an autoencoder that shrinks pictures 16 times in each direction (into 64 channels), supports 2K pictures natively, and distills the model, which its authors run for 40 steps, into one that makes comparable pictures in 4 network evaluations. None of this replaces the loop you just ran: predict, estimate the clean image, step, repeat.