Using diffusion models in practice

Generating an image is a loop that runs a fixed number of times, and nearly every practical decision is about that loop. This section covers the controls you actually have: how many steps to run and what running fewer costs you, how hard to push the output towards the prompt before it goes stiff, and the ways of conditioning on something other than text when a prompt cannot describe what you want. It ends on adapting a model to a style, and on the fact that most of the engineering in a real system sits around the model rather than in it.

Steps and samplers#

The reverse process was originally many small steps. Better samplers reach comparable quality in far fewer, which is where most of the practical speedup since has come from. Step count is the main quality-versus-latency dial.

~50 steps    high quality, slow
~20 steps    close, much faster, the usual default
~4 steps     distilled models, real quality cost

Latency scales roughly linearly with steps, so this
is the first thing to tune when generation is too slow.

The sampler is the setting behind that dial, and it is chosen at generation time from the same trained weights. A DDPM-style loop that simply runs every twentieth step falls apart, because its update assumes a single small move; a DDIM-style sampler, which converts the noise prediction into an estimate of the finished image and re-noises it to whatever level comes next, takes the same jump without a quality loss you can measure. In the 2-D experiment in the diffusion maths topic, running every 50th DDPM step scored 0.4323 on a distance-to-data metric where pure noise scores 0.4810, while DDIM at the same 20 steps scored 0.0744 against 0.0717 for the full 1,000-step loop. That is the whole story, and it is the reason to change the sampler before the step count.

The names you will meet are DDIM, DPM-Solver and its multistep and second-order variants, Euler and Euler ancestral, and UniPC. They differ in how cleverly they extrapolate between the levels they visit, and the practical ordering is dull: solver-based samplers such as DPM-Solver++ and UniPC are good at 15 to 25 steps, Euler ancestral is stochastic so it will not reproduce across runs, and DDIM is the deterministic baseline every implementation has. Ancestral samplers add fresh noise at each step, which means a fixed seed does not give you a fixed image if you change the step count — a reproducibility trap worth knowing before you build an evaluation grid.

Guidance#

Classifier-free guidance controls how strongly the output follows the prompt, by contrasting the conditioned and unconditioned predictions and pushing away from the latter. Low values drift from the prompt; high values produce oversaturated, rigid images. There is a usable middle and it is worth finding empirically.

What the scale actually trades is measurable, and on a toy problem where every quantity can be computed exactly it comes out like this: adherence to the condition saturates almost immediately, diversity falls steadily, and fidelity holds flat and then collapses. In that experiment every sample was already on the requested class by a scale of 2, while the spread between samples fell eightfold between scale 1 and scale 5, and distance-to-data was flat from 2 to 10 before jumping twentyfold by scale 20. Map that onto what you see in images and it explains the familiar pattern exactly: raising guidance past the point where the prompt is being followed does not buy more adherence, it buys less variety and eventually the saturated, plastic look, which is samples leaving the region real images occupy.

Two consequences for practice. Distilled few-step models are usually trained with the guidance already baked in, so they want a guidance scale of 0 or 1 and will produce garbage at the 7 you use elsewhere — read the model card for the intended value rather than carrying yours over. And because guidance costs a second forward pass per step, turning it off is not a small saving: it halves the work.

What generation actually costs#

Latency and price are the same calculation, and the only measurement you need is one forward pass of the network at your resolution. Everything else is multiplication, and doing it before you choose settings is what stops a pilot becoming an unaffordable product.

unet_ms = 55       # one forward pass at your resolution, measured on your own hardware

for steps, guided in ((50, True), (20, True), (8, True), (4, False)):
    passes = steps * (2 if guided else 1)
    latency = passes * unet_ms / 1000
    print(f"{steps:3d} steps, guidance {'on ' if guided else 'off'}: "
          f"{passes:3d} passes  {latency:5.2f} s/image  {3600/latency:7.0f} images/GPU-hour")
#  50 steps, guidance on : 100 passes   5.50 s/image      655 images/GPU-hour
#  20 steps, guidance on :  40 passes   2.20 s/image     1636 images/GPU-hour
#   8 steps, guidance on :  16 passes   0.88 s/image     4091 images/GPU-hour
#   4 steps, guidance off:   4 passes   0.22 s/image    16364 images/GPU-hour

gpu_hour = 2.50    # dollars, whatever you are actually paying

for steps, guided in ((50, True), (20, True), (4, False)):
    passes = steps * (2 if guided else 1)
    per_image = passes * unet_ms / 1000 / 3600 * gpu_hour
    print(f"{steps:3d} steps: {per_image*100:5.2f} cents/image, "
          f"{per_image*100_000:7.0f} dollars per 100k images")
#  50 steps:  0.38 cents/image,     382 dollars per 100k images
#  20 steps:  0.15 cents/image,     153 dollars per 100k images
#   4 steps:  0.02 cents/image,      15 dollars per 100k images

Conditioning on more than text#

A prompt is a poor way to say where things go. When a user wants this pose, this layout, or this face, the answer is almost never a longer prompt; it is conditioning the process on something other than text. These are the three that cover most real requirements.

• Image-to-image: start from a noisy version of an existing image rather than pure noise, so the output keeps its structure.
• Inpainting: fix part of the image and generate only the masked region.
• Structural conditioning: guide generation with an edge map, depth map or pose, which gives far more control than a prompt alone.

Image-to-image has one number that decides everything and is usually misunderstood: the strength, which sets how far up the noise schedule you start. At strength 0.3 you noise the input to about a third of the way and run only the remaining steps, so the composition survives and the style changes; at 0.9 almost nothing of the original is left and the prompt is effectively generating from scratch. It also means image-to-image is cheaper than text-to-image in proportion to the strength: strength 0.3 with a 20-step setting runs 6 steps, not 20. Structural conditioning, which is what ControlNet-style adapters do, is the opposite trade: it costs an extra network evaluation per step but gives you geometry a prompt cannot express.

Adapting a model#

Full fine-tuning is expensive. LoRA adapters work here as they do for language models and are the common approach for adding a style or a subject. Textual inversion learns a new token embedding for a concept without touching the model, which is cheaper still and less flexible.

method              trains              typical data   artefact size   composes
prompt engineering  nothing             0 images       0               n/a
textual inversion   one token embedding 3-5 images     a few KB        yes
LoRA                low-rank adapters   10-50 images   10-200 MB       yes, several
full fine-tune      every weight        1000+ images   the whole model no

Two things decide whether an adapter works, and neither is the training code. The first is the captions: a subject LoRA trained on twenty photographs captioned identically will learn the background and the lighting along with the subject, so caption what varies and leave out what should be baked in. The second is knowing what you are adapting — a LoRA trained against one base model does not transfer to another, and the commonest support question in this area is somebody loading an adapter built for a different base and reporting that the model is broken.

What to skip#

You do not need to know the derivation behind each sampler to choose one, and you will not be asked. Skip the taxonomy of solver orders, skip memorising which scheduler pairs with which model, and skip the prompt-keyword folklore entirely — the lists of quality words that circulate were fitted to one base model and mostly do nothing on another. What is worth holding is the cost arithmetic above, the fact that guidance is two passes, the strength parameter for image-to-image, and the ordering of adaptation methods.