Diffusion & generative models

Image, video and audio generation is nearly all done by diffusion models now, and they work by a route that sounds backwards: learn to take noise away, then start from nothing but noise. This section covers how that training works, why it is so much steadier than the generative adversarial networks it displaced, how a text prompt steers it, and the price it pays, which is that one sample takes many passes through the model instead of one.

The idea#

Take an image and add a little noise. Repeat until it is pure noise. That is the forward process and it requires no learning. Now train a model to reverse one step: given a noisy image, predict the noise that was added. Apply it repeatedly starting from pure noise and you generate an image.

forward   image -> slightly noisy -> ... -> pure noise     (fixed)
reverse   pure noise -> ... -> slightly noisy -> image    (learned)

The forward process is one line of arithmetic#

The word process makes the noising sound like something you have to run. You do not. Adding Gaussian noise a thousand times is itself adding Gaussian noise once, with a known scale, so there is a closed form for the noisy image at any step. Call the surviving fraction of the signal alphabar_t. Then x_t = sqrt(alphabar_t) times the clean image plus sqrt(1 - alphabar_t) times a fresh draw of standard noise. A training example is a random image, a random t and one line.

import torch

T = 1000
betas = torch.linspace(1e-4, 0.02, T)      # the schedule: how much noise each step adds
alphas = 1 - betas
abar = torch.cumprod(alphas, dim=0)        # how much signal survives steps 1..t

for t in (49, 199, 499, 799, 999):
    print(f"t={t+1:4d}  alphabar={abar[t]:.6f}  "
          f"signal x{abar[t].sqrt():.4f}  noise x{(1-abar[t]).sqrt():.4f}")
# t=  50  alphabar=0.971016  signal x0.9854  noise x0.1702
# t= 200  alphabar=0.659039  signal x0.8118  noise x0.5839
# t= 500  alphabar=0.078587  signal x0.2803  noise x0.9599
# t= 800  alphabar=0.001532  signal x0.0391  noise x0.9992
# t=1000  alphabar=0.000040  signal x0.0064  noise x1.0000

It is worth checking that the shortcut is really equal to the long way round, because everything downstream depends on it. Noising 200,000 copies of the value 0.8 one step at a time for 500 steps gives mean 0.2222 and standard deviation 0.9617; the one-line version gives 0.2253 and 0.9622, against a predicted 0.2243 and 0.9599. Same distribution, 500 times cheaper.

What the model predicts, and why it is the noise#

Given x_t you could ask the network for either of two things: the clean image x_0, or the noise that was mixed in. They are algebraically the same request, because the forward equation rearranges to x_0 = (x_t - sqrt(1 - alphabar_t) * eps) / sqrt(alphabar_t). Knowing one gives you the other exactly. What differs is how hard the target is to hit at each noise level, and that turns out to matter.

At t = 1000 the signal multiplier is 0.0064. Asking for x_0 there is asking the network to reconstruct an image from something that retains 0.6 per cent of it: the best any model can do is output the dataset average, and the squared error is enormous. Asking for the noise at the same step is nearly free, because x_t is almost exactly the noise. The noise target is well scaled at every t, so the loss is not dominated by the steps where the task is impossible. Training the same small network on the same data with the two targets, and sampling from each, gives a mean distance from generated points to the real data of 0.0767 for noise prediction and 0.0975 for image prediction — the same model, a quarter worse, for choosing the harder target.

Why it beat GANs#

A GAN trains a generator against a discriminator, and that adversarial setup is famously unstable: mode collapse, oscillation, sensitivity to hyperparameters. Diffusion has a simple regression objective, predict the noise, which trains stably and scales predictably. Stability is why the field moved.

Stability is a claim you can measure rather than assert, and the way to measure it is to vary something you should not have to think about and see whether the result survives. Below, a small GAN and a small diffusion model are trained on the same two-crescent dataset across four learning rates, two Adam beta settings and three seeds each — 48 runs — and scored by the mean distance from a generated point to the nearest real one, where the data itself scores about 0.011.

   lr  beta1     GAN, three seeds          diffusion, three seeds
0.0003    0.5   0.112  0.080  0.058        0.110  0.107  0.114
0.0003    0.9   0.098  0.145  0.124        0.110  0.108  0.110
0.0010    0.5   0.046  0.042  0.045        0.074  0.080  0.083
0.0010    0.9   0.067  0.115  0.100        0.082  0.081  0.084
0.0030    0.5   0.017  0.019  0.027        0.035  0.055  0.058
0.0030    0.9   0.042  0.129  0.070        0.046  0.059  0.055
0.0100    0.5   0.020  0.079  0.080        0.065  0.060  0.068
0.0100    0.9   0.392  4.181  0.059        0.032  0.037  0.046

Read the last row first. At a learning rate of 0.01 with Adam's first beta at 0.9, one GAN seed scored 4.181 — eight times worse than generating pure noise, which is what divergence looks like — while another seed at the identical setting scored 0.059, among its better results. The diffusion model's three seeds at that same setting were 0.032, 0.037 and 0.046, its best cell in the table. Nothing in the GAN's loss curve tells you in advance which of those two outcomes you are about to get.

The honest reading is not that GANs are bad. In its good configurations the GAN beat diffusion outright, 0.017 against 0.035, and a GAN samples in one forward pass rather than dozens. The problem is that the hyperparameter search is part of the method, and the search is not guided by anything you can see while it runs. Diffusion's worst cell across all 24 runs is within a factor of two of its best, and no run failed. On a two-dimensional toy that is a mild inconvenience. At image scale, where one configuration is days of cluster time and a failed seed is discovered at the end, it is the entire argument, and it is why the field moved.

Conditioning#

Text-to-image works by conditioning each denoising step on a text embedding, so the model is predicting the noise given both the noisy image and the description. Classifier-free guidance strengthens the influence of the text by contrasting conditioned and unconditioned predictions.

The mechanism is one line and no extra model. During training the condition is dropped at random, typically 10 per cent of the time, and replaced by a null token, so the same network learns both the conditional and the unconditional prediction. At sampling time you evaluate both and extrapolate: eps = eps_uncond + w * (eps_cond - eps_uncond). At w = 1 that is exactly the conditional prediction. Above 1 you are pushing further along the direction the prompt moved the answer than the model itself would have gone, which is why guidance is described as a strength rather than a switch, and why it costs two forward passes per step.

Latent diffusion, and why it was the unlock#

Latent diffusion, which made this practical at consumer scale, runs the process in a compressed latent space rather than on pixels, so each step is far cheaper. The size of the saving is worth doing on paper once, because it explains why text-to-image went from a research cluster to a laptop in about a year.

H = W = 512
pixel = H * W * 3
latent = (H // 8) * (W // 8) * 4     # a downsample of 8 into 4 channels, the usual VAE

print(f"pixel space : {H}x{W}x3 = {pixel:,} values")
print(f"latent space: {H//8}x{W//8}x4 = {latent:,} values")
print(f"{pixel/latent:.0f}x fewer values per denoising step")
print(f"attention pairs: {(H*W)**2:,} in pixels vs {((H//8)*(W//8))**2:,} in latents")
# pixel space : 512x512x3 = 786,432 values
# latent space: 64x64x4 = 16,384 values
# 48x fewer values per denoising step
# attention pairs: 68,719,476,736 in pixels vs 16,777,216 in latents

The cost is that the autoencoder is now part of the model's ceiling. Anything it cannot reconstruct, the diffusion model cannot generate, which is why small text and fine regular patterns were bad in early latent models and why later versions widened the latent from four channels to sixteen. If you are debugging poor fine detail, encode and decode a real image without any diffusion at all and look at what comes back; if the detail is already gone there, no amount of sampling will bring it back.

What to skip#

The variational derivation of the training objective — the evidence lower bound, the term-by-term reduction to a weighted squared error — is a page of algebra that ends at the loss stated above, and nobody will ask you to reproduce it. Read it once if you enjoy that, and skip the score-matching and stochastic-differential-equation formulations entirely until you need them; they are a different set of words for the same model. What you are expected to hold is the closed-form forward process, the noise-prediction loss, one sampling step, and the guidance formula. Those four fit on an index card and carry every question in this area.

Generating an image is a loop that runs a fixed number of times, and nearly every practical decision is about that loop. This section covers the controls you actually have: how many steps to run and what running fewer costs you, how hard to push the output towards the prompt before it goes stiff, and the ways of conditioning on something other than text when a prompt cannot describe what you want. It ends on adapting a model to a style, and on the fact that most of the engineering in a real system sits around the model rather than in it.

Steps and samplers#

The reverse process was originally many small steps. Better samplers reach comparable quality in far fewer, which is where most of the practical speedup since has come from. Step count is the main quality-versus-latency dial.

~50 steps    high quality, slow
~20 steps    close, much faster, the usual default
~4 steps     distilled models, real quality cost

Latency scales roughly linearly with steps, so this
is the first thing to tune when generation is too slow.

The sampler is the setting behind that dial, and it is chosen at generation time from the same trained weights. A DDPM-style loop that simply runs every twentieth step falls apart, because its update assumes a single small move; a DDIM-style sampler, which converts the noise prediction into an estimate of the finished image and re-noises it to whatever level comes next, takes the same jump without a quality loss you can measure. In the 2-D experiment in the diffusion maths topic, running every 50th DDPM step scored 0.4323 on a distance-to-data metric where pure noise scores 0.4810, while DDIM at the same 20 steps scored 0.0744 against 0.0717 for the full 1,000-step loop. That is the whole story, and it is the reason to change the sampler before the step count.

The names you will meet are DDIM, DPM-Solver and its multistep and second-order variants, Euler and Euler ancestral, and UniPC. They differ in how cleverly they extrapolate between the levels they visit, and the practical ordering is dull: solver-based samplers such as DPM-Solver++ and UniPC are good at 15 to 25 steps, Euler ancestral is stochastic so it will not reproduce across runs, and DDIM is the deterministic baseline every implementation has. Ancestral samplers add fresh noise at each step, which means a fixed seed does not give you a fixed image if you change the step count — a reproducibility trap worth knowing before you build an evaluation grid.

Guidance#

Classifier-free guidance controls how strongly the output follows the prompt, by contrasting the conditioned and unconditioned predictions and pushing away from the latter. Low values drift from the prompt; high values produce oversaturated, rigid images. There is a usable middle and it is worth finding empirically.

What the scale actually trades is measurable, and on a toy problem where every quantity can be computed exactly it comes out like this: adherence to the condition saturates almost immediately, diversity falls steadily, and fidelity holds flat and then collapses. In that experiment every sample was already on the requested class by a scale of 2, while the spread between samples fell eightfold between scale 1 and scale 5, and distance-to-data was flat from 2 to 10 before jumping twentyfold by scale 20. Map that onto what you see in images and it explains the familiar pattern exactly: raising guidance past the point where the prompt is being followed does not buy more adherence, it buys less variety and eventually the saturated, plastic look, which is samples leaving the region real images occupy.

Two consequences for practice. Distilled few-step models are usually trained with the guidance already baked in, so they want a guidance scale of 0 or 1 and will produce garbage at the 7 you use elsewhere — read the model card for the intended value rather than carrying yours over. And because guidance costs a second forward pass per step, turning it off is not a small saving: it halves the work.

What generation actually costs#

Latency and price are the same calculation, and the only measurement you need is one forward pass of the network at your resolution. Everything else is multiplication, and doing it before you choose settings is what stops a pilot becoming an unaffordable product.

unet_ms = 55       # one forward pass at your resolution, measured on your own hardware

for steps, guided in ((50, True), (20, True), (8, True), (4, False)):
    passes = steps * (2 if guided else 1)
    latency = passes * unet_ms / 1000
    print(f"{steps:3d} steps, guidance {'on ' if guided else 'off'}: "
          f"{passes:3d} passes  {latency:5.2f} s/image  {3600/latency:7.0f} images/GPU-hour")
#  50 steps, guidance on : 100 passes   5.50 s/image      655 images/GPU-hour
#  20 steps, guidance on :  40 passes   2.20 s/image     1636 images/GPU-hour
#   8 steps, guidance on :  16 passes   0.88 s/image     4091 images/GPU-hour
#   4 steps, guidance off:   4 passes   0.22 s/image    16364 images/GPU-hour

gpu_hour = 2.50    # dollars, whatever you are actually paying

for steps, guided in ((50, True), (20, True), (4, False)):
    passes = steps * (2 if guided else 1)
    per_image = passes * unet_ms / 1000 / 3600 * gpu_hour
    print(f"{steps:3d} steps: {per_image*100:5.2f} cents/image, "
          f"{per_image*100_000:7.0f} dollars per 100k images")
#  50 steps:  0.38 cents/image,     382 dollars per 100k images
#  20 steps:  0.15 cents/image,     153 dollars per 100k images
#   4 steps:  0.02 cents/image,      15 dollars per 100k images

Conditioning on more than text#

A prompt is a poor way to say where things go. When a user wants this pose, this layout, or this face, the answer is almost never a longer prompt; it is conditioning the process on something other than text. These are the three that cover most real requirements.

• Image-to-image: start from a noisy version of an existing image rather than pure noise, so the output keeps its structure.
• Inpainting: fix part of the image and generate only the masked region.
• Structural conditioning: guide generation with an edge map, depth map or pose, which gives far more control than a prompt alone.

Image-to-image has one number that decides everything and is usually misunderstood: the strength, which sets how far up the noise schedule you start. At strength 0.3 you noise the input to about a third of the way and run only the remaining steps, so the composition survives and the style changes; at 0.9 almost nothing of the original is left and the prompt is effectively generating from scratch. It also means image-to-image is cheaper than text-to-image in proportion to the strength: strength 0.3 with a 20-step setting runs 6 steps, not 20. Structural conditioning, which is what ControlNet-style adapters do, is the opposite trade: it costs an extra network evaluation per step but gives you geometry a prompt cannot express.

Adapting a model#

Full fine-tuning is expensive. LoRA adapters work here as they do for language models and are the common approach for adding a style or a subject. Textual inversion learns a new token embedding for a concept without touching the model, which is cheaper still and less flexible.

method              trains              typical data   artefact size   composes
prompt engineering  nothing             0 images       0               n/a
textual inversion   one token embedding 3-5 images     a few KB        yes
LoRA                low-rank adapters   10-50 images   10-200 MB       yes, several
full fine-tune      every weight        1000+ images   the whole model no

Two things decide whether an adapter works, and neither is the training code. The first is the captions: a subject LoRA trained on twenty photographs captioned identically will learn the background and the lighting along with the subject, so caption what varies and leave out what should be baked in. The second is knowing what you are adapting — a LoRA trained against one base model does not transfer to another, and the commonest support question in this area is somebody loading an adapter built for a different base and reporting that the model is broken.

What to skip#

You do not need to know the derivation behind each sampler to choose one, and you will not be asked. Skip the taxonomy of solver orders, skip memorising which scheduler pairs with which model, and skip the prompt-keyword folklore entirely — the lists of quality words that circulate were fitted to one base model and mostly do nothing on another. What is worth holding is the cost arithmetic above, the fact that guidance is two passes, the strength parameter for image-to-image, and the ordering of adaptation methods.

Diffusion models are trained by destroying data with noise and learning to undo it. This section is the four equations that say exactly that, with every symbol explained: the schedule of noise, the closed form that lets you jump to any noise level in one line, the training loss, which is plain squared error on the noise that was added, and a single step of sampling. The closed form is the piece everything else rests on, because it is why training is cheap: you never simulate a thousand noising steps to make one training example.

A schedule of small noise steps#

beta_t        small amount of noise added at step t, e.g. from 0.0001 up to 0.02 over T = 1000 steps
alpha_t       1 - beta_t, how much signal survives step t
alphabar_t    alpha_1 x alpha_2 x ... x alpha_t, how much signal survives all steps up to t

One step of the forward process is x_t = sqrt(alpha_t) * x_(t-1) + sqrt(beta_t) * z. The square roots are there so that variance is preserved: if x_(t-1) has unit variance, alpha_t of it survives, beta_t of fresh noise is added, and alpha_t + beta_t = 1, so x_t has unit variance too. That is the whole reason the schedule is written in terms of beta rather than a plain noise standard deviation, and it is why data is normalised to roughly zero mean and unit variance before any of this starts.

Skip to any step directly#

x_t = sqrt(alphabar_t) * x_0  +  sqrt(1 - alphabar_t) * noise        noise ~ Normal(0, 1)

Because adding Gaussian noise repeatedly is still Gaussian, you do not need to run t noising steps to get x_t. One line produces a noisy version of a clean image at any level. This is what makes training cheap: pick a random image, a random t, and noise it in one go.

Do not take that on trust. Running the chain 500 times on 200,000 copies of a single value and comparing with the one-line version is four lines of code, and it is the check that catches a wrong schedule before it costs you a training run.

import torch
torch.manual_seed(0)

T = 1000
betas = torch.linspace(1e-4, 0.02, T)
alphas = 1 - betas
abar = torch.cumprod(alphas, dim=0)

N, t_check, x0 = 200_000, 500, 0.8
x = torch.full((N,), x0)
for t in range(t_check):                                   # the long way
    x = alphas[t].sqrt() * x + betas[t].sqrt() * torch.randn(N)
print(f"chain, {t_check} steps : mean {x.mean():.4f}  std {x.std():.4f}")

a = abar[t_check - 1]                                      # the one-line way
y = a.sqrt() * x0 + (1 - a).sqrt() * torch.randn(N)
print(f"closed form, 1 line  : mean {y.mean():.4f}  std {y.std():.4f}")
print(f"what it should be    : mean {a.sqrt()*x0:.4f}  std {(1-a).sqrt():.4f}")
# chain, 500 steps : mean 0.2222  std 0.9617
# closed form, 1 line  : mean 0.2253  std 0.9622
# what it should be    : mean 0.2243  std 0.9599

The numbers the schedule actually produces are worth having in your head, because they are what makes the reverse process possible at all. With the linear schedule above, alphabar is 0.971 at t = 50, 0.659 at t = 200, 0.0786 at t = 500 and 0.00004 at t = 1000. So at the last step the original image is multiplied by 0.0064 — it is gone, to any precision that matters, which is exactly what lets sampling begin from a fresh draw of noise that has never seen a dataset.

The training objective#

loss = mean( (noise - model(x_t, t))^2 )

The model sees the noisy image and the step number and predicts the noise that was mixed in. It is plain regression with a squared error. If you know the noise, you know the clean image, since the equation above can be solved for x_0.

x_0 = ( x_t  -  sqrt(1 - alphabar_t) * eps )  /  sqrt(alphabar_t)

That scaling is the whole argument for the noise target. At t = 1000, sqrt(alphabar_t) is 0.0064, so recovering x_0 means dividing by 0.0064 — amplifying the model's error by a factor of 156. A network asked for x_0 directly at that step can do no better than output the dataset mean, and the squared error there swamps the loss from every step where the task was learnable. Predicting the noise keeps the target at unit scale for every t, so no part of the schedule dominates. Trained identically on the same data, noise prediction beat image prediction by about a quarter on sample quality in the 2-D experiment in the next section.

The full derivation from the variational bound puts a t-dependent weight in front of this squared error. The result that made diffusion practical is that dropping the weight entirely and using the plain mean squared error above works better in practice. So the loss you implement is simpler than the loss the theory hands you, and knowing that saves you looking for the weighting term in someone's code and concluding you have misread it.

One sampling step#

x_(t-1) = ( x_t - beta_t / sqrt(1 - alphabar_t) * model(x_t, t) ) / sqrt(alpha_t)  +  sqrt(beta_t) * z

z is fresh noise (none on the final step)

Subtract the predicted noise, scaled for this step, rescale, and add a little new noise. Repeat from pure noise down to step one. Faster samplers take bigger, smarter steps using the same trained model.

Two details in that line trip people up. The first is that the coefficient on the predicted noise is beta_t / sqrt(1 - alphabar_t) and not sqrt(1 - alphabar_t): you are removing only the noise this step added, not all the noise accumulated so far, so the subtraction is small. At t = 500 that coefficient is 0.0105, against a total accumulated noise scale of 0.96. Each step nudges. The second is the fresh noise z added back afterwards. Removing it makes the sampler deterministic and, with the DDPM formula, makes the samples worse, because the update is an approximation of a distribution rather than a point and the variance is part of what is being modelled. Deterministic sampling is a real option, but it needs the DDIM update rather than this one.

That DDIM update is worth writing down alongside, because it is what every fast sampler is a variant of. Predict the noise, convert it to an estimate of the clean image, then re-noise that estimate to the next noise level you want — which need not be t - 1. It is the freedom to choose the next level that lets a sampler run 20 steps instead of 1000.

x0_hat  = ( x_t - sqrt(1 - alphabar_t) * model(x_t, t) ) / sqrt(alphabar_t)
x_prev  = sqrt(alphabar_prev) * x0_hat  +  sqrt(1 - alphabar_prev) * model(x_t, t)

alphabar_prev is the noise level of the next step you have chosen, not necessarily t - 1

Classifier-free guidance#

guided = model(x_t, t, empty_prompt) + w * ( model(x_t, t, prompt) - model(x_t, t, empty_prompt) )

Train one model that sometimes sees the prompt and sometimes an empty one. At sampling time, take the direction the prompt pushes the prediction, and exaggerate it by a factor w. It costs two model evaluations per step.

Read the endpoints of that formula. At w = 0 the prompt term cancels and you get the unconditional model: a sample from the data distribution, ignoring what you asked for. At w = 1 the two unconditional terms cancel and you get the plain conditional model, which is what an honest sample from p(image given prompt) looks like. Every value above 1 is extrapolation past the model's own conditional prediction, which is why the usual range of 5 to 8 is not a mild setting: the samples produced there are not draws from the conditional distribution at all. They are draws from a sharpened version of it, which is exactly why they look better and exactly why diversity falls.

The name is historical and actively misleading. The original method, classifier guidance, ran a separately trained classifier on the noisy image and pushed the sample along the gradient of the class score. Classifier-free guidance gets the same effect from one model trained with the condition dropped at random, typically 10 per cent of the time, and needs no classifier. Nothing in the formula above involves a classifier; the word only records what it replaced.

Where the field went next#

Several recent image models use flow matching, a closely related formulation that learns a velocity carrying noise to data along straighter paths, which allows fewer sampling steps. The intuitions here carry over directly.

Concretely, flow matching interpolates linearly between noise and data, x_t = (1 - t) * x_0 + t * noise with t running from 0 to 1, and trains the network to predict the velocity noise - x_0 rather than the noise alone. Sampling is then integrating an ordinary differential equation, and because the paths are close to straight, a crude integrator with few steps gets close to the answer. The forward process, the regression objective and guidance all keep their shape; what changes is the interpolation and what the network is asked to output. If you have understood the four equations above you can read a flow-matching paper the same afternoon.

You have the equations; this is running them. The dataset is a few thousand points arranged in two crescents on a plane, which trains in about a minute on a laptop and lets you watch structure appear out of noise in a scatter plot. Everything here transfers to images unchanged except the network. Starting in two dimensions is deliberate: on images a diffusion bug looks exactly like needing more training for several hours, while on points it is visible immediately.

Start in two dimensions#

Images take hours to train. Points on a plane take a minute on a laptop CPU and you can see every step. Everything here transfers to images; only the network changes.

import torch, torch.nn as nn
from sklearn.datasets import make_moons
torch.manual_seed(0)

X = torch.tensor(make_moons(10_000, noise=0.05, random_state=0)[0], dtype=torch.float32)

T = 1000
betas = torch.linspace(1e-4, 0.02, T)
alphas = 1 - betas
abar = torch.cumprod(alphas, dim=0)

class Denoiser(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(3, 128), nn.SiLU(),
            nn.Linear(128, 128), nn.SiLU(),
            nn.Linear(128, 2),
        )

    def forward(self, x, t):                        # x: [n, 2], t: [n]
        return self.net(torch.cat([x, (t / T).unsqueeze(-1)], dim=-1))

Train#

model = Denoiser()
opt = torch.optim.Adam(model.parameters(), lr=1e-3)

for step in range(10_000):
    x0 = X[torch.randint(len(X), (256,))]
    t = torch.randint(T, (256,))
    noise = torch.randn_like(x0)
    a = abar[t].unsqueeze(-1)
    xt = a.sqrt() * x0 + (1 - a).sqrt() * noise
    loss = ((model(xt, t.float()) - noise) ** 2).mean()
    opt.zero_grad(); loss.backward(); opt.step()
    if step % 2000 == 0 or step == 9999:
        print(f"step {step:5d}  loss {loss.item():.4f}")

# step     0  loss 1.0144
# step  2000  loss 0.1892
# step  4000  loss 0.2067
# step  6000  loss 0.1869
# step  8000  loss 0.2063
# step  9999  loss 0.1632     all 10,000 steps in a few seconds on a laptop CPU

The loss will not go to zero and should not. At large t the model is being asked to predict a noise vector from something that is almost entirely that noise vector, which is easy; at small t it is being asked to identify which specific noise was added to a nearly clean point, which is close to impossible, and the irreducible error there sets the floor. A converged run on this dataset sits around 0.16 to 0.20. The number itself tells you very little — that is why the next part measures the samples instead.

Sample#

@torch.no_grad()
def sample(n=2000):
    x = torch.randn(n, 2)
    for t in reversed(range(T)):
        eps = model(x, torch.full((n,), float(t)))
        x = (x - betas[t] / (1 - abar[t]).sqrt() * eps) / alphas[t].sqrt()
        if t > 0:
            x = x + betas[t].sqrt() * torch.randn_like(x)
    return x

points = sample()      # scatter plot these: two interleaved moons should appear

Score it, do not squint at it#

A scatter plot will tell you whether the crescents are there. It will not tell you whether a change helped, and by the third experiment you will be comparing plots from memory. One number fixes that: for each generated point, the distance to the nearest real point, averaged. It is cheap, it has an obvious floor, and it has an obvious ceiling, so a result is interpretable without any further apparatus.

real = X[torch.randperm(len(X))[:2000]]
fit = lambda p: torch.cdist(p, real).min(dim=1).values.mean().item()

print(f"generated  {fit(sample()):.4f}")
print(f"real data  {fit(X[torch.randperm(len(X))[:2000]]):.4f}")   # the floor
print(f"noise      {fit(torch.randn(2000, 2)):.4f}")               # the ceiling
# generated  0.0758
# real data  0.0107
# noise      0.4810

This metric only rewards being close to the data, so a model that produced two thousand copies of the same real point would score a perfect 0. Pair it with a spread measure — the mean distance from each generated point to its nearest other generated point — before you trust it. That pairing is the whole of generative evaluation in miniature: one number for fidelity, one for coverage, and neither is meaningful alone. Reported means and standard deviations agree too: generated points came out at mean (0.436, 0.286) with standard deviations (0.902, 0.496), against (0.500, 0.250) and (0.867, 0.498) for the data.

Fewer steps, and why the sampler is not a detail#

The obvious way to make sampling ten times faster is to run every tenth step of the same loop. Try it and the result is not ten times faster at slightly worse quality; it is ten times faster at no quality. The DDPM update assumes you are moving one step, and its coefficients are wrong for a jump. Swapping in the DDIM update, which converts the noise prediction into an estimate of the clean point and re-noises it to whatever level you ask for next, fixes it with no retraining at all.

@torch.no_grad()
def ddim(n, steps):
    ts = torch.linspace(T - 1, 0, steps).long().tolist()
    x = torch.randn(n, 2)
    for i, t in enumerate(ts):
        eps = model(x, torch.full((n,), float(t)))
        a = abar[t]
        x0 = (x - (1 - a).sqrt() * eps) / a.sqrt()               # estimate the clean point
        a_prev = abar[ts[i + 1]] if i + 1 < len(ts) else torch.tensor(1.0)
        x = a_prev.sqrt() * x0 + (1 - a_prev).sqrt() * eps       # re-noise to the next level
    return x

steps    DDPM loop, every kth step    DDIM
 1000        0.0717  (0.87 s)          0.0766  (0.67 s)
  100        0.3098  (0.07 s)          0.0787  (0.07 s)
   50        0.3933  (0.04 s)          0.0765  (0.11 s)
   20        0.4323  (0.02 s)          0.0744  (0.02 s)
   10        0.4608  (0.01 s)          0.0819  (0.01 s)

pure noise scores 0.4810, real data scores 0.0107

That table is the reason sampler names appear in every image-generation interface and why people argue about them. It is also the reason a diffusion result is not reproducible without saying which sampler produced it. If you take one habit from this section, take this: when generation is too slow, change the sampler before you change anything else, and measure rather than assume.

Conditioning and guidance, measured#

make_moons returns a label saying which crescent each point came from, so the same dataset supports conditional generation. Add an embedding for the label, reserve one extra index for no label, and drop the real label 10 per cent of the time during training. That single change is all classifier-free guidance requires: one network that has learned both the conditional and the unconditional prediction.

Xn, Yn = make_moons(10_000, noise=0.05, random_state=0)   # this time keep the labels
X = torch.tensor(Xn, dtype=torch.float32)
Y = torch.tensor(Yn, dtype=torch.long)
real = X[torch.randperm(len(X))[:2000]]

class Cond(nn.Module):
    def __init__(self):
        super().__init__()
        self.emb = nn.Embedding(3, 16)          # labels 0 and 1, plus 2 meaning no label
        self.net = nn.Sequential(
            nn.Linear(3 + 16, 128), nn.SiLU(),
            nn.Linear(128, 128), nn.SiLU(),
            nn.Linear(128, 2),
        )

    def forward(self, x, t, y):
        return self.net(torch.cat([x, (t / T).unsqueeze(-1), self.emb(y)], dim=-1))

cond = Cond()
opt = torch.optim.Adam(cond.parameters(), lr=1e-3)
for step in range(20_000):
    i = torch.randint(len(X), (256,))
    x0, y = X[i], Y[i].clone()
    y[torch.rand(256) < 0.1] = 2                # drop the label 10% of the time
    t = torch.randint(T, (256,))
    noise = torch.randn_like(x0)
    a = abar[t].unsqueeze(-1)
    xt = a.sqrt() * x0 + (1 - a).sqrt() * noise
    loss = ((cond(xt, t.float(), y) - noise) ** 2).mean()
    opt.zero_grad(); loss.backward(); opt.step()

@torch.no_grad()
def guided(n, cls, w, steps=100):
    ts = torch.linspace(T - 1, 0, steps).long().tolist()
    x = torch.randn(n, 2)
    for i, t in enumerate(ts):
        tt = torch.full((n,), float(t))
        eps_uncond = cond(x, tt, torch.full((n,), 2))
        eps_cond   = cond(x, tt, torch.full((n,), cls))
        eps = eps_uncond + w * (eps_cond - eps_uncond)          # the whole of guidance
        a = abar[t]
        x0 = (x - (1 - a).sqrt() * eps) / a.sqrt()
        a_prev = abar[ts[i + 1]] if i + 1 < len(ts) else torch.tensor(1.0)
        x = a_prev.sqrt() * x0 + (1 - a_prev).sqrt() * eps
    return x

Now the guidance scale stops being folklore. Train that model for 20,000 steps, sample 2,000 points asking for crescent 0 at each of several values of w, and score three things: what fraction of the samples a nearest-neighbour classifier assigns to the crescent you asked for, the fit-to-data number from above, and the spread.

   w   on-class   fit-to-data   spread
 0.0      57.4%        0.0753   0.0230     unconditional: it ignores what you asked for
 1.0      99.5%        0.0189   0.0119     the honest conditional model
 2.0     100.0%        0.0115   0.0046
 3.0     100.0%        0.0107   0.0026
 5.0     100.0%        0.0109   0.0014     8x less varied than at w = 1
10.0     100.0%        0.0130   0.0014
20.0     100.0%        0.2340   0.0017     off the data entirely
40.0     100.0%        0.7531   0.0066     worse than pure noise, which scores 0.4810

Three things fall out of that table that are usually stated as opinions. First, guidance buys adherence to the condition almost immediately: by w = 2 every sample is already on the requested crescent, so the adherence argument for a higher scale is finished there. Second, what continues past that point is the collapse in diversity, a factor of eight between w = 1 and w = 5, which on images is why heavily guided prompts return near-identical compositions across seeds. Third, fidelity holds for a while and then goes suddenly: fit is flat from w = 2 to w = 10 and then jumps to 0.2340 at w = 20 and 0.7531 at w = 40, worse than pure noise, because extrapolating that far past the conditional prediction pushes samples off the data entirely. On images that same failure is the oversaturated, high-contrast, plastic look, and now you know it is not a stylistic quirk of the model but points leaving the region the data occupies.

Things to try#

• Plot intermediate x at several t to watch structure appear from noise.
• Sample with only 100 of the 1,000 steps by skipping, then with DDIM at 100 steps. The gap is the table above, reproduced on your own run.
• Train a second model with the identical loop but the target set to x0 instead of noise, and compare fit-to-data. Noise prediction came out at 0.0767 against 0.0975 for image prediction.
• Break the schedule deliberately: set betas to linspace(1e-4, 0.002) so alphabar at T is 0.37 rather than 0.00004, retrain, and watch sampling fail because the starting noise is nothing like the end of the forward process.
• Move to small images such as MNIST with a small convolutional network.

Most image generation work is not training a model but running someone else's well. This section covers a diffusers pipeline from a few lines of code through the settings that actually change the output: the sampler, which can be swapped without retraining, and the step count and guidance scale, which trade quality against latency. The method that matters is measuring rather than eyeballing, with a fixed grid of prompts, seeds and settings, plus the two checks before anything ships, which are the licence and what the safety filter does and does not catch.

A pipeline in a few lines#

import torch
from diffusers import AutoPipelineForText2Image

pipe = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/sdxl-turbo", torch_dtype=torch.float16, variant="fp16"
).to("cuda")

image = pipe("a lighthouse on a cliff at dusk, watercolour",
             num_inference_steps=1, guidance_scale=0.0).images[0]
image.save("lighthouse.png")

The sampler is swappable#

from diffusers import DPMSolverMultistepScheduler

pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)

The trained model predicts noise; the scheduler decides how to step. Changing it changes speed and look without retraining.

That separation is the single most useful thing to understand about the library. A pipeline is a bag of independently loaded components — a text encoder and its tokenizer, the denoising network, the variational autoencoder that moves between pixels and latents, and a scheduler — and the scheduler is the only one that holds no weights. Swapping it is free and reversible. Swapping anything else means loading different weights, and mixing components trained against different bases is the cause of most inexplicably bad output people report.

Measure, do not eyeball#

Pick twenty prompts that represent your use case and render a grid across step counts and guidance values with fixed seeds. Choose from the grid, and record latency for each cell. Casual testing on one prompt picks settings that fail on others.

The reason to be formal about this is that the choice is a three-way trade — quality, latency and cost — and eyeballing optimises the one you happened to be looking at. Build the grid as data, not as a contact sheet, and score it with something you wrote down before you looked.

import itertools

prompts = [f"prompt {i}" for i in range(20)]      # your own, representative of real use
seeds = [0, 1, 2]
steps = [8, 20, 30, 50]
scales = [3.0, 5.0, 7.5]

cells = list(itertools.product(steps, scales))
renders = len(cells) * len(prompts) * len(seeds)
unet_ms = 55                                       # measure one forward pass yourself
seconds = sum(s * 2 * unet_ms / 1000 for s, _ in cells) * len(prompts) * len(seeds)

print(f"{len(cells)} settings x {len(prompts)} prompts x {len(seeds)} seeds = {renders} images")
print(f"about {seconds/60:.0f} minutes of GPU time for the whole grid")
for s, g in cells:
    print(f"  steps={s:2d} scale={g:3.1f}  {s*2*unet_ms/1000:.2f} s/image")
# 12 settings x 20 prompts x 3 seeds = 720 images
# about 36 minutes of GPU time for the whole grid
#   steps= 8 scale=3.0  0.88 s/image
#   steps=20 scale=3.0  2.20 s/image
#   steps=30 scale=3.0  3.30 s/image
#   steps=50 scale=3.0  5.50 s/image

Score each cell on three things and keep them separate: whether the image matches the prompt, whether it is free of the failures you care about, and what it cost. The first two need human judgement on a sample or an automated proxy you have checked against human judgement on that same sample — a CLIP-style image-text similarity score is the usual proxy for prompt adherence and it is only worth using once you have confirmed it agrees with you on fifty pairs. Then pick the cheapest cell that is not distinguishably worse than the best one, which is almost never the highest step count.

Before a product#

• Check the model licence. Some restrict commercial use, and terms differ between versions.
• Add input and output safety filtering. Pipelines often include a checker; know what it does and does not catch.
• Plan for cost: images per second per GPU at your chosen settings decides the bill.

The licence point is the one that ends projects rather than delaying them, and it is not uniform across a family: successive releases of the same model line have shipped under different terms, including ones that gate commercial use behind a subscription or a revenue threshold. Read the licence file in the repository for the exact revision you are pulling, record the revision hash in your deployment, and do not infer the terms from a sibling model. The same care applies to any LoRA or adapter you load, which carries its own licence and often its own training-data provenance question.

On safety filtering, the built-in checkers are classifiers on the generated image with a fixed threshold, and they are neither the beginning nor the end of the problem. They do not see the prompt, so they cannot catch a request that is harmful in intent but innocuous as pixels; they are tuned for one category of content and miss others entirely; and they produce false positives that your users will experience as the product being broken. Filter on the input as well as the output, log what was blocked so you can measure the false-positive rate, and decide deliberately what happens on a block, because silently returning a blank image is the worst of the options.

What to skip#

Skip collecting prompt-keyword lists, skip the argument about which scheduler looks best in the abstract, and skip fine-tuning until you have a measured grid showing the base model cannot do the thing. What is worth your time is the measurement harness, because it is the part that keeps working when the model underneath you is replaced, which on current timescales is about twice a year.