Working with pretrained image models

Most image generation work is not training a model but running someone else's well. This section covers a diffusers pipeline from a few lines of code through the settings that actually change the output: the sampler, which can be swapped without retraining, and the step count and guidance scale, which trade quality against latency. The method that matters is measuring rather than eyeballing, with a fixed grid of prompts, seeds and settings, plus the two checks before anything ships, which are the licence and what the safety filter does and does not catch.

A pipeline in a few lines#

import torch
from diffusers import AutoPipelineForText2Image

pipe = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/sdxl-turbo", torch_dtype=torch.float16, variant="fp16"
).to("cuda")

image = pipe("a lighthouse on a cliff at dusk, watercolour",
             num_inference_steps=1, guidance_scale=0.0).images[0]
image.save("lighthouse.png")

The sampler is swappable#

from diffusers import DPMSolverMultistepScheduler

pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)

The trained model predicts noise; the scheduler decides how to step. Changing it changes speed and look without retraining.

That separation is the single most useful thing to understand about the library. A pipeline is a bag of independently loaded components — a text encoder and its tokenizer, the denoising network, the variational autoencoder that moves between pixels and latents, and a scheduler — and the scheduler is the only one that holds no weights. Swapping it is free and reversible. Swapping anything else means loading different weights, and mixing components trained against different bases is the cause of most inexplicably bad output people report.

Measure, do not eyeball#

Pick twenty prompts that represent your use case and render a grid across step counts and guidance values with fixed seeds. Choose from the grid, and record latency for each cell. Casual testing on one prompt picks settings that fail on others.

The reason to be formal about this is that the choice is a three-way trade — quality, latency and cost — and eyeballing optimises the one you happened to be looking at. Build the grid as data, not as a contact sheet, and score it with something you wrote down before you looked.

import itertools

prompts = [f"prompt {i}" for i in range(20)]      # your own, representative of real use
seeds = [0, 1, 2]
steps = [8, 20, 30, 50]
scales = [3.0, 5.0, 7.5]

cells = list(itertools.product(steps, scales))
renders = len(cells) * len(prompts) * len(seeds)
unet_ms = 55                                       # measure one forward pass yourself
seconds = sum(s * 2 * unet_ms / 1000 for s, _ in cells) * len(prompts) * len(seeds)

print(f"{len(cells)} settings x {len(prompts)} prompts x {len(seeds)} seeds = {renders} images")
print(f"about {seconds/60:.0f} minutes of GPU time for the whole grid")
for s, g in cells:
    print(f"  steps={s:2d} scale={g:3.1f}  {s*2*unet_ms/1000:.2f} s/image")
# 12 settings x 20 prompts x 3 seeds = 720 images
# about 36 minutes of GPU time for the whole grid
#   steps= 8 scale=3.0  0.88 s/image
#   steps=20 scale=3.0  2.20 s/image
#   steps=30 scale=3.0  3.30 s/image
#   steps=50 scale=3.0  5.50 s/image

Score each cell on three things and keep them separate: whether the image matches the prompt, whether it is free of the failures you care about, and what it cost. The first two need human judgement on a sample or an automated proxy you have checked against human judgement on that same sample — a CLIP-style image-text similarity score is the usual proxy for prompt adherence and it is only worth using once you have confirmed it agrees with you on fifty pairs. Then pick the cheapest cell that is not distinguishably worse than the best one, which is almost never the highest step count.

Before a product#

• Check the model licence. Some restrict commercial use, and terms differ between versions.
• Add input and output safety filtering. Pipelines often include a checker; know what it does and does not catch.
• Plan for cost: images per second per GPU at your chosen settings decides the bill.

The licence point is the one that ends projects rather than delaying them, and it is not uniform across a family: successive releases of the same model line have shipped under different terms, including ones that gate commercial use behind a subscription or a revenue threshold. Read the licence file in the repository for the exact revision you are pulling, record the revision hash in your deployment, and do not infer the terms from a sibling model. The same care applies to any LoRA or adapter you load, which carries its own licence and often its own training-data provenance question.

On safety filtering, the built-in checkers are classifiers on the generated image with a fixed threshold, and they are neither the beginning nor the end of the problem. They do not see the prompt, so they cannot catch a request that is harmful in intent but innocuous as pixels; they are tuned for one category of content and miss others entirely; and they produce false positives that your users will experience as the product being broken. Filter on the input as well as the output, log what was blocked so you can measure the false-positive rate, and decide deliberately what happens on a block, because silently returning a blank image is the worst of the options.

What to skip#

Skip collecting prompt-keyword lists, skip the argument about which scheduler looks best in the abstract, and skip fine-tuning until you have a measured grid showing the base model cannot do the thing. What is worth your time is the measurement harness, because it is the part that keeps working when the model underneath you is replaced, which on current timescales is about twice a year.