Generating by removing noise

Image, video and audio generation is nearly all done by diffusion models now, and they work by a route that sounds backwards: learn to take noise away, then start from nothing but noise. This section covers how that training works, why it is so much steadier than the generative adversarial networks it displaced, how a text prompt steers it, and the price it pays, which is that one sample takes many passes through the model instead of one.

The idea#

Take an image and add a little noise. Repeat until it is pure noise. That is the forward process and it requires no learning. Now train a model to reverse one step: given a noisy image, predict the noise that was added. Apply it repeatedly starting from pure noise and you generate an image.

forward   image -> slightly noisy -> ... -> pure noise     (fixed)
reverse   pure noise -> ... -> slightly noisy -> image    (learned)

The forward process is one line of arithmetic#

The word process makes the noising sound like something you have to run. You do not. Adding Gaussian noise a thousand times is itself adding Gaussian noise once, with a known scale, so there is a closed form for the noisy image at any step. Call the surviving fraction of the signal alphabar_t. Then x_t = sqrt(alphabar_t) times the clean image plus sqrt(1 - alphabar_t) times a fresh draw of standard noise. A training example is a random image, a random t and one line.

import torch

T = 1000
betas = torch.linspace(1e-4, 0.02, T)      # the schedule: how much noise each step adds
alphas = 1 - betas
abar = torch.cumprod(alphas, dim=0)        # how much signal survives steps 1..t

for t in (49, 199, 499, 799, 999):
    print(f"t={t+1:4d}  alphabar={abar[t]:.6f}  "
          f"signal x{abar[t].sqrt():.4f}  noise x{(1-abar[t]).sqrt():.4f}")
# t=  50  alphabar=0.971016  signal x0.9854  noise x0.1702
# t= 200  alphabar=0.659039  signal x0.8118  noise x0.5839
# t= 500  alphabar=0.078587  signal x0.2803  noise x0.9599
# t= 800  alphabar=0.001532  signal x0.0391  noise x0.9992
# t=1000  alphabar=0.000040  signal x0.0064  noise x1.0000

It is worth checking that the shortcut is really equal to the long way round, because everything downstream depends on it. Noising 200,000 copies of the value 0.8 one step at a time for 500 steps gives mean 0.2222 and standard deviation 0.9617; the one-line version gives 0.2253 and 0.9622, against a predicted 0.2243 and 0.9599. Same distribution, 500 times cheaper.

What the model predicts, and why it is the noise#

Given x_t you could ask the network for either of two things: the clean image x_0, or the noise that was mixed in. They are algebraically the same request, because the forward equation rearranges to x_0 = (x_t - sqrt(1 - alphabar_t) * eps) / sqrt(alphabar_t). Knowing one gives you the other exactly. What differs is how hard the target is to hit at each noise level, and that turns out to matter.

At t = 1000 the signal multiplier is 0.0064. Asking for x_0 there is asking the network to reconstruct an image from something that retains 0.6 per cent of it: the best any model can do is output the dataset average, and the squared error is enormous. Asking for the noise at the same step is nearly free, because x_t is almost exactly the noise. The noise target is well scaled at every t, so the loss is not dominated by the steps where the task is impossible. Training the same small network on the same data with the two targets, and sampling from each, gives a mean distance from generated points to the real data of 0.0767 for noise prediction and 0.0975 for image prediction — the same model, a quarter worse, for choosing the harder target.

Why it beat GANs#

A GAN trains a generator against a discriminator, and that adversarial setup is famously unstable: mode collapse, oscillation, sensitivity to hyperparameters. Diffusion has a simple regression objective, predict the noise, which trains stably and scales predictably. Stability is why the field moved.

Stability is a claim you can measure rather than assert, and the way to measure it is to vary something you should not have to think about and see whether the result survives. Below, a small GAN and a small diffusion model are trained on the same two-crescent dataset across four learning rates, two Adam beta settings and three seeds each — 48 runs — and scored by the mean distance from a generated point to the nearest real one, where the data itself scores about 0.011.

   lr  beta1     GAN, three seeds          diffusion, three seeds
0.0003    0.5   0.112  0.080  0.058        0.110  0.107  0.114
0.0003    0.9   0.098  0.145  0.124        0.110  0.108  0.110
0.0010    0.5   0.046  0.042  0.045        0.074  0.080  0.083
0.0010    0.9   0.067  0.115  0.100        0.082  0.081  0.084
0.0030    0.5   0.017  0.019  0.027        0.035  0.055  0.058
0.0030    0.9   0.042  0.129  0.070        0.046  0.059  0.055
0.0100    0.5   0.020  0.079  0.080        0.065  0.060  0.068
0.0100    0.9   0.392  4.181  0.059        0.032  0.037  0.046

Read the last row first. At a learning rate of 0.01 with Adam's first beta at 0.9, one GAN seed scored 4.181 — eight times worse than generating pure noise, which is what divergence looks like — while another seed at the identical setting scored 0.059, among its better results. The diffusion model's three seeds at that same setting were 0.032, 0.037 and 0.046, its best cell in the table. Nothing in the GAN's loss curve tells you in advance which of those two outcomes you are about to get.

The honest reading is not that GANs are bad. In its good configurations the GAN beat diffusion outright, 0.017 against 0.035, and a GAN samples in one forward pass rather than dozens. The problem is that the hyperparameter search is part of the method, and the search is not guided by anything you can see while it runs. Diffusion's worst cell across all 24 runs is within a factor of two of its best, and no run failed. On a two-dimensional toy that is a mild inconvenience. At image scale, where one configuration is days of cluster time and a failed seed is discovered at the end, it is the entire argument, and it is why the field moved.

Conditioning#

Text-to-image works by conditioning each denoising step on a text embedding, so the model is predicting the noise given both the noisy image and the description. Classifier-free guidance strengthens the influence of the text by contrasting conditioned and unconditioned predictions.

The mechanism is one line and no extra model. During training the condition is dropped at random, typically 10 per cent of the time, and replaced by a null token, so the same network learns both the conditional and the unconditional prediction. At sampling time you evaluate both and extrapolate: eps = eps_uncond + w * (eps_cond - eps_uncond). At w = 1 that is exactly the conditional prediction. Above 1 you are pushing further along the direction the prompt moved the answer than the model itself would have gone, which is why guidance is described as a strength rather than a switch, and why it costs two forward passes per step.

Latent diffusion, and why it was the unlock#

Latent diffusion, which made this practical at consumer scale, runs the process in a compressed latent space rather than on pixels, so each step is far cheaper. The size of the saving is worth doing on paper once, because it explains why text-to-image went from a research cluster to a laptop in about a year.

H = W = 512
pixel = H * W * 3
latent = (H // 8) * (W // 8) * 4     # a downsample of 8 into 4 channels, the usual VAE

print(f"pixel space : {H}x{W}x3 = {pixel:,} values")
print(f"latent space: {H//8}x{W//8}x4 = {latent:,} values")
print(f"{pixel/latent:.0f}x fewer values per denoising step")
print(f"attention pairs: {(H*W)**2:,} in pixels vs {((H//8)*(W//8))**2:,} in latents")
# pixel space : 512x512x3 = 786,432 values
# latent space: 64x64x4 = 16,384 values
# 48x fewer values per denoising step
# attention pairs: 68,719,476,736 in pixels vs 16,777,216 in latents

The cost is that the autoencoder is now part of the model's ceiling. Anything it cannot reconstruct, the diffusion model cannot generate, which is why small text and fine regular patterns were bad in early latent models and why later versions widened the latent from four channels to sixteen. If you are debugging poor fine detail, encode and decode a real image without any diffusion at all and look at what comes back; if the detail is already gone there, no amount of sampling will bring it back.

What to skip#

The variational derivation of the training objective — the evidence lower bound, the term-by-term reduction to a weighted squared error — is a page of algebra that ends at the loss stated above, and nobody will ask you to reproduce it. Read it once if you enjoy that, and skip the score-matching and stochastic-differential-equation formulations entirely until you need them; they are a different set of words for the same model. What you are expected to hold is the closed-form forward process, the noise-prediction loss, one sampling step, and the guidance formula. Those four fit on an index card and carry every question in this area.