AI modelsDive 06 of 06

Diffusion models, from noise to an image

Turn noise into structure with a real diffusion sampler, then follow an image through denoising, guidance, latent patches and the GPU arithmetic behind every step.

Levels
9
Reading
42 min
Depth
Idea to Machine
Published

A distribution takes shape

No image is hiding in the noise.

Each dot starts as a Gaussian draw. An exact denoising field carries it toward one of eight clusters.

Starting rule
x ∼ N(0, I)
Update rule
Follow the score
Destination
A new data sample
Try the particle sampler
Computed with 96 deterministic DDIM steps. The field is analytic; an image model learns to approximate such a field in many more dimensions.

Level 01/09Idea

Make generation a denoising problem

You ask for a coral mug on a wooden table, morning light. The program draws random numbers. It runs a neural network on them, changes the numbers a little, and repeats. Eventually a decoder turns the result into pixels: a mug, a handle, a table, a shadow. The first array contained none of those objects. The trained network supplied the structure.

The difficult part of image generation is that pixels have to agree. A dark patch might be coffee, a shadow, or the inside of a handle. Its meaning depends on distant patches. Generating each pixel independently gives static. Predicting all pixels with a squared-error loss from a prompt alone tends toward an average of many plausible images. A blurred average of ten possible mugs is rarely a convincing mug.

A A generative model trained to reverse a process that gradually corrupts data with noise. changes the prediction problem. Take a real image, add a known amount of random noise, and teach a network to predict the corruption. Repeat at many noise levels. At generation time, start with almost pure noise and use those predictions to take a sequence of steps toward the data distribution. Every step asks the same network a local question: given this array and this noise level, which correction is useful?

This turns generation into supervised regression with training targets you can manufacture. It does not make learning images trivial. At heavy noise, the network must infer broad structure from very weak evidence; at light noise, it must model texture and edges. But the objective is direct, and the network can share knowledge across all noise levels. You do not need a separate human label for the noise in every image.

A small distribution you can see completely

We will keep the mug as the image example and use a second, fully inspectable example for the mathematics: eight fuzzy clusters on a ring. Each data sample has just two coordinates. Each cluster is a Gaussian with standard deviation about 0.212, and its centre is 1.8 units from the origin. An equal-weight mixture chooses one of those eight clusters, then draws a point inside it.

Because this distribution is known, we can calculate the correct denoising direction without training. The hero above and the particle lab below use that calculation. The browser evaluates the mixture’s analytic score and applies the sampling updates directly. For photographs, the density is unknown and enormously higher-dimensional, so a neural network learns the field we can calculate here.

How we got here

Sohl-Dickstein and colleagues introduced diffusion probabilistic models in 2015. Ho, Jain and Abbeel’s DDPM paper in 2020 made a particularly effective connection between image generation and predicting Gaussian noise. Score-based models developed a closely related view: learn a vector field pointing toward higher probability at each noise level.

FamilyWhat training asks forWhat generation runs
GANFool a learned discriminatorA generator, often in one pass
DiffusionPredict noise, clean data, or a related targetRepeated denoising updates
Flow matchingPredict a velocity along chosen probability pathsA numerical ODE solver
Three ways to turn randomness into samples. Diffusion replaces an adversarial training signal with denoising targets; flow matching makes the transport path an explicit design choice.

This history is a branching tree, rather than a sequence of methods that made their predecessors obsolete. GANs remain useful; diffusion can sample deterministically; flow matching can use diffusion-like paths. What changes is the combination of training objective, probability path, network architecture and sampler. Keeping those four choices separate will make the controls of an image generator much less mysterious.

There is also no requirement to generate one pixel after another. During a denoising step, the network works on the whole noisy image or its compressed representation. Spatial positions exchange information through convolutions or attention. The next step gets a revised whole array. That parallelism is useful on a GPU, while the dependency between successive steps makes the number of steps expensive.

Learn how data behaves at every noise level, then use that knowledge to turn a simple random draw into a structured sample.

Level 02/09Interface

What the controls actually control

Before opening the training equations, pin down the interface. Imagine that two runs use the same checkpoint and the same prompt about our coral mug. One has 8 steps and guidance 1; the other has 40 steps and guidance 7. They are asking the same learned network to generate with different numerical procedures and different conditioning strength. Neither run trains the checkpoint again.

The prompt becomes a sequence of text embeddings. The seed initializes a pseudorandom number generator. The schedule chooses which noise levels to visit. The sampler decides how to travel between them. The step count limits how often that travel consults the network. Resolution fixes the size of the array on which almost all of this work happens.

ControlChangesTry it here
PromptThe condition supplied to the networkThe guidance lab uses “right-hand clusters” as its condition
SeedInitial noise and any later random drawsNext seed gives another sample from the same target
StepsNumerical resolution of samplingCompare 8 with 128 in the particle sampler
Guidance scaleStrength of the conditional correctionMove w from 0 to 8 in the guidance lab
SamplerUpdate rule, including whether it adds noiseCompare ancestral DDPM and deterministic DDIM below
ScheduleNoise levels and where computation is spentInspect linear and cosine signal-to-noise curves
Negative promptA condition used for the reference guidance branchSee the guidance equation and its interpretation
ResolutionLatent dimensions and token countDouble the edge in the cost calculator
The controls describe different parts of one generation. A reproducible run records the checkpoint and software configuration along with these values.

A seed is a recipe for randomness

A seed does not name a picture. It names a sequence of pseudorandom numbers within a particular implementation. Keep the checkpoint, shapes, software, sampler and execution settings fixed, and it is useful for repeatable comparisons. Change the image dimensions and the random tensor changes shape. Change the sampler and later noise draws may be consumed differently. Across libraries or GPU kernels, equal seed numbers alone do not guarantee identical pixels.

The particle labs make repeatability deliberately narrow and exact: the same seed, settings and JavaScript arithmetic reproduce the same coordinate arrays. Both sampler panels begin with identical particles. The ancestral panel draws additional noise during sampling; the deterministic panel does not. This lets you compare their behaviour without mistaking a different initial draw for an algorithmic difference.

Starting partway down the noise ladder

The same machinery can edit an existing image. Encode it, add noise up to a selected time, then denoise with a new condition. A low starting noise level preserves much of the original structure because the sampler has only a short distance to travel. A high level gives the model more freedom because more of the input information has been destroyed. The control often called “strength” usually determines this starting point and the portion of the inference grid that remains.

For the mug, a weak edit might change its colour while preserving the handle position. A stronger edit can change its shape or composition. This is a statistical tendency, rather than a boundary around permitted changes. Spatial masks and other conditioning mechanisms can make edits more selective, but they introduce additional contracts beyond the plain text-conditioned sampler. Notice that this editing workflow needs the VAE encoder at runtime, while generation from pure noise does not.

Why a larger number can make the result worse

More steps usually reduce a numerical approximation error, until model error or other limits dominate. Guidance changes the field itself. Turning guidance up is therefore a different kind of intervention from buying more steps. A strongly guided result may follow the prompt more visibly while losing variety, becoming unnaturally saturated, or developing artifacts. There is no universal best scale across checkpoints and samplers.

“Scheduler” is especially overloaded in image software. One API may use it for the forward training noise schedule; another may package the inference timestep grid and solver into the same object. Ask two concrete questions: what noise level is the model receiving now, and which equation produces the next array? The forward schedule and the inference grid are related, but they are not the same list of settings.

The controls also need to fit the model. A distilled model trained for a few large jumps can need a different guidance convention or sampler from its teacher. A velocity-predicting model cannot be interpreted as a noise-predicting one without conversion. These are mathematical contracts between weights and runtime, rather than cosmetic labels in a dropdown.

Check yourselfYou keep the seed but switch from DDIM to an ancestral sampler. Must you get the same result?

Level 03/09Runtime

Destroy an image, exactly

Let x₀ be a clean image represented by real numbers, usually scaled or normalized before training. Let t run from 1 to T. At step t, choose a small noise variance βₜ, and write αₜ = 1 − βₜ. The forward Markov process shrinks the previous array and adds fresh independent Gaussian noise:

q(xₜ | xₜ₋₁) = N(√αₜ xₜ₋₁, βₜ I)

I is the identity matrix: before any model processes it, the injected noise has independent coordinates. This is a rule we choose, so no neural network is needed to run it. After one step, some signal remains. After enough steps, the original image contributes almost nothing and the distribution is close to a standard Gaussian.

Running all previous steps every time you want a training example would be wasteful. Gaussian sums give a shortcut. Define the cumulative product ᾱₜ = ∏ᵢ₌₁ᵗ αᵢ, with ᾱ₀ = 1. Then any forward marginal has a closed form:

q(xₜ | x₀) = N(√ᾱₜ x₀, (1 − ᾱₜ)I)
xₜ = √ᾱₜ x₀ + √(1 − ᾱₜ) ε,   ε ∼ N(0, I)

That equation gives a training pair with a single random tensor. Pick the time, pick ε, mix them with the clean image, and you know exactly which noise you used. The slider below applies it to the mug’s actual pixel values. Move to the clean endpoint, then increase time until the handle disappears into the noise.

Clean · x₀

Corrupted · xₜ

Seed 7

96 × 96 scalar values in [−1, 1]. Display clips and maps them into the coral palette. One fixed ε per seed makes this a marginal interpolation, not a Markov trajectory.

Forward noising of a procedurally drawn mug. The displayed coefficients are the actual signal and noise multipliers used for every pixel.

Why the square roots matter

ᾱₜ measures the squared signal multiplier. The noise variance is 1 − ᾱₜ, so the noise standard deviation is its square root. Mixing the two arrays with weights ᾱₜ and 1 − ᾱₜ instead would produce a different process. At ᾱₜ = 0.5, the two correct amplitude coefficients are both about 0.707, rather than 0.5.

The resulting image may contain values outside the clean image range. Training uses those real values; a display has to map or clip them to screen intensities. The demo clips only when writing the canvas. At the noisiest end, you see a display of Gaussian values, not a claim that Gaussian noise has bounded pixel intensities.

Spend time where signal remains

The Signal variance relative to the injected noise variance; here defined for the normalized forward process as ᾱₜ / (1 − ᾱₜ). is SNRₜ = ᾱₜ / (1 − ᾱₜ). This convention measures the schedule coefficients; the empirical variance of a particular dataset can add a scale factor. At high SNR, most structure is visible. At low SNR, the network must use learned statistical structure to infer a correction.

A linear β schedule does not make signal strength fall linearly. It multiplies many shrinking factors. With T = 1000 and β increasing from 0.0001 to 0.02, ᾱ falls rapidly through the middle of the process. The cosine schedule introduced by Nichol and Dhariwal preserves signal longer before approaching the noisy endpoint. Their schedule uses f(t) = cos²(((t/T + s)/(1 + s))π/2), ᾱₜ = f(t)/f(0), and s = 0.008.

  • Linear β · solid
  • Cosine ᾱ · dashed

Signal variance ᾱₜ · from 1 to 0

Clean · 0Noise · 1

log₁₀ SNR · clipped to [−4, 4]

Clean · 0Noise · 1

Linear: T = 1000, β₁ = 0.0001, βₜ = 0.02 at t = T. Cosine: offset 0.008; endpoint floored at 10⁻⁸ for numerical sampling.

Equal forward times can represent very different prediction problems. Scrub the vertical cursor to compare signal variance and SNR under the two schedules.

The clean endpoint has infinite SNR, so the graph clips its logarithmic axis. A practical discrete cosine schedule also caps the final β to avoid a singular transition. Our analytic lab floors ᾱ at 10⁻⁸ instead, starts from a standard Gaussian approximation, and uses the displayed continuous cosine curve between grid points.

The schedule matters twice: it defines the corrupted examples during training, and it tells an inference runtime how to interpret its current array. You can choose fewer inference times from a compatible noise parameterization, but you must still tell the network the correct noise level. Feeding it the wrong time is like asking a thermometer to interpret Celsius readings as Fahrenheit.

The forward process is chosen arithmetic. Its closed form supplies unlimited noisy inputs with known regression targets.

Level 04/09Runtime

Learn a direction at every noise level

Now hide ε from the network and ask it to predict it. The input is xₜ, the time t, and optionally a condition c such as text. The output εθ(xₜ, t, c) has the same shape as the noisy image. θ names the trainable weights. The widely used simplified DDPM objective is a mean squared error:

Lε = Eₓ₀,ₜ,ε [ ‖ε − εθ(xₜ, t, c)‖² ]

Sample clean examples, times and noise independently, construct xₜ with the closed form, and regress against ε. The expectation means averaging across those draws; implementations also choose how to average over coordinates. The training dive explains how the resulting loss produces gradients, how backpropagation computes them, and how an optimizer updates θ. Here, the important choice is the target those gradients teach.

Noise prediction · one training batch, PyTorch-like pseudocode
# A single training batch; x0 is scaled image or latent data.
t = randint(1, T + 1, (batch_size,))
epsilon = randn_like(x0)
a = alpha_bar[t].reshape(batch_size, 1, 1, 1)
xt = sqrt(a) * x0 + sqrt(1 - a) * epsilon
prediction = denoiser(xt, t, condition)
loss = mean((prediction - epsilon) ** 2)
loss.backward()
optimizer.step()
optimizer.zero_grad()

This simple loss is related to the variational likelihood bound in DDPM, with a different weighting across times. Objectives that look like the same squared error can emphasize different noise levels through weighting or sampling. They need not optimize exactly the same trade-off between likelihood, perceptual quality and numerical behaviour.

The answer is an expectation, not the hidden random draw

Given one corrupted array, many combinations of clean image and noise are possible. The network cannot uniquely recover the particular random tensor used by the training code. Under squared error, the optimal predictor is the conditional mean E[ε | xₜ, c]. At high noise it averages over many possible explanations. As sampling proceeds, the current sample changes, and later predictions become compatible with one increasingly specific image.

This explains why “remove all the predicted noise at once” does not usually generate a sharp image. A clean-image estimate from a very noisy input is an average over possibilities. Sampling repeatedly moves the input distribution so that those possibilities separate. It is the evolving distribution of samples, not a single early denoised preview, that carries the generative process.

Noise prediction is also score prediction

The The gradient of log probability density with respect to the current sample coordinates, rather than with respect to model weights. is sₜ(x) = ∇ₓ log pₜ(x). It points toward increasing log density at that noise level. For the Gaussian forward corruption, the optimal noise predictor and the score are related by:

sθ(xₜ, t, c) = −εθ(xₜ, t, c) / √(1 − ᾱₜ)

The gradient here is with respect to the input coordinates. A trained noise predictor provides it through its output; generation does not backpropagate through the model’s weights. Song and colleagues’ score-based SDE framework uses such a time-dependent score to construct reverse stochastic dynamics and deterministic probability-flow dynamics.

Tweedie’s denoising identity makes the connection concrete. For ᾱₜ > 0, a correct marginal score gives the posterior mean clean sample:

E[x₀ | xₜ] = (xₜ + (1 − ᾱₜ)sₜ(xₜ)) / √ᾱₜ

Our mixture makes the score especially simple. At time t, cluster k has mean √ᾱₜ μₖ and scalar variance τₜ² = ᾱₜ · 0.045 + 1 − ᾱₜ. The responsibility rₖ(x, t) is the posterior probability that a point at x came from cluster k. Evaluate each noised Gaussian at x, multiply by its prior weight of 1/8, and normalize across clusters. At high noise, many clusters are plausible; near a clean cluster, one usually dominates. The score is the responsibility-weighted mean displacement divided by τₜ²:

sₜ(x) = Σₖ rₖ(x, t)(√ᾱₜ μₖ − x) / τₜ²

Computing rₖ requires only exponentials and normalized weights. The lab evaluates them with a log-sum-exp stabilization, so points far from the clusters do not underflow all the weights to zero. A photograph model replaces this known-density calculation with a learned approximation.

Noise predictionεθ ≈ E[ε | xₜ]

Predict the added noise.

Data predictionx̂₀ ≈ E[x₀ | xₜ]

Predict the clean sample.

v-predictionv = √ᾱₜ ε − √(1 − ᾱₜ)x₀

Predict a rotated target.

Three prediction targets describe the same Gaussian corruption. Converting between them requires the same cumulative ᾱ used to construct the input.

To see the rotation, treat x₀ and ε as two coordinate axes. The forward input is one linear combination; v is the orthogonal combination. Given a v prediction, x̂₀ = √ᾱₜ xₜ − √(1 − ᾱₜ)vθ and ε̂ = √(1 − ᾱₜ)xₜ + √ᾱₜ vθ. These conversions avoid dividing by √ᾱₜ when recovering x̂₀. Salimans and Ho discuss this parameterization in progressive distillation.

The ability to convert targets algebraically does not make their training losses identical. Squared error on v implies a different effective weighting of errors on ε across noise levels. And this v is not automatically the velocity along every possible flow-matching path. A path and a time convention must be specified before “velocity” has a definite meaning.

Check yourselfAt very high noise, what does an ideal squared-error noise predictor recover?

Level 05/09Runtime

Choose the path, predict its velocity

The Gaussian diffusion path is one way to connect data with noise. Suppose instead that you draw one noise point z and one data point x₀ and connect them with a straight line. Introduce a new time u that increases from noise at 0 to data at 1. This direction is the reverse of the forward-noising t used above.

xᵤ = (1 − u)z + ux₀
d xᵤ / du = x₀ − z

For that particular pair, the target velocity is constant. To train a model, sample a pair and a time, compute the interpolated input, and regress against the pair’s displacement. This requires no numerical ODE simulation inside the training loop. The target comes straight from the selected endpoints.

Linear conditional flow matching · noise-to-data time convention
# u increases from noise (0) to data (1).
u = rand((batch_size, 1, 1, 1))
noise = randn_like(x0)
xu = (1 - u) * noise + u * x0
target_velocity = x0 - noise
prediction = velocity_model(xu, u, condition)
loss = mean((prediction - target_velocity) ** 2)

Flow matching, introduced by Lipman and colleagues, provides a general framework for learning vector fields along chosen probability paths. Its choices extend beyond straight interpolation. Rectified flow, from Liu, Gong and Liu, learns transport using straight endpoint paths and can re-pair endpoints through a learned flow to encourage straighter transport on subsequent training rounds.

Again, the network does not receive the endpoint pair. It receives xᵤ and u. If many lines pass through the same neighbourhood at the same time, their velocities disagree. The squared-error optimum is their conditional mean, bθ(xᵤ, u) ≈ E[x₀ − z | xᵤ, u]. That averaged field transports the marginal distribution. Integrating it can give curved trajectories even though each conditional training line is straight.

  • Gaussian diffusion coupling · curved
  • Linear endpoint interpolation · straight

DDIM trajectories · 128 updates

Linear training paths

Both panels share endpoints for comparison. The straight lines illustrate conditional training paths; a learned marginal flow can curve and need not produce this same endpoint coupling.

The same noise and data endpoints connected two ways. Scrub to compare a deterministic diffusion trajectory with straight endpoint interpolation.

Why path geometry affects step count

A numerical solver sees a vector field and has to follow it using finitely many evaluations. If the direction changes sharply between two evaluations, one long straight update cuts across the curve. A path with nearly constant velocity is easier to integrate accurately in few steps. That is a reason to seek straighter transports, but a straight training interpolation alone does not guarantee a one-step generator.

Even in one dimension, the distribution in the middle of an interpolation depends on which endpoints were paired. Pairing each noise point independently with a data point produces a different training problem from pairing them using an existing transport map. Endpoint distributions can be the same while the geometry and difficulty between them differ. Reflow uses this freedom to improve the coupling.

Diffusion and flow models share much of their runtime machinery. Both can use a transformer backbone, text conditioning, latent representations and numerical solvers. A deterministic probability-flow ODE even lets a diffusion score define a transport velocity. The distinction is best made by inspecting the training path and prediction target, rather than assuming that any iterative image model must be a DDPM.

A named image-model example

Esser and colleagues’ 2024 Stable Diffusion 3 paper studies rectified-flow training, timestep sampling and a multimodal diffusion transformer, MMDiT. It is a useful public example of the ingredients being separable: latent compression sets the representation, the flow objective sets the regression task, and MMDiT sets how image and text tokens interact.

“Diffusion” in a product name is therefore not enough to identify the training equations. For the rest of this dive, the Gaussian equations refer to the DDPM-style process explicitly. The flow example keeps its own u and bθ notation. That distinction also matters when selecting a scheduler: a runtime must match the model’s time, scaling and target conventions.

Training chooses a path between distributions. Sampling pays for how hard the resulting learned field is to integrate.

Level 06/09Runtime

Walk back from noise

A score tells you which direction increases density. It does not, by itself, tell you how to generate the right distribution. Repeated gradient ascent would herd samples toward density peaks and erase their spread. A diffusion sampler combines the score with the drift, scaling and noise required by the chosen reverse process.

For a DDPM Gaussian reverse transition, a common noise-parameterized update is:

xₜ₋₁ = (xₜ − βₜ εθ(xₜ, t) / √(1 − ᾱₜ)) / √αₜ + σₜz

Here z is a fresh standard Gaussian draw. With the fixed posterior-variance choice, σₜ² = β̃ₜ = βₜ(1 − ᾱₜ₋₁)/(1 − ᾱₜ). Other DDPM variants fix or learn the variance differently. At the final update this choice has zero variance. The sequence is called ancestral sampling because it draws each next state from the learned conditional transition.

Small reverse steps are well approximated by Gaussians, but a long reverse jump can have multiple plausible destinations. Even a perfect score does not make the Gaussian approximation exact at arbitrarily coarse step counts. The 2-D lab lets you isolate that numerical and transition error from the difficulty of training an image model.

DDPM ancestral · η = 1

DDIM deterministic · η = 0

Exact mixture scores, 256 particles, cosine schedule. Circles show two standard deviations of the target clusters. Finite steps introduce sampling error.

A real reverse-diffusion lab with an analytic score. The panels start with the same Gaussian particles; traces show the first eight paths. Compare 8 and 128 steps, then replay or scrub the trajectories.

The ancestral panel uses the posterior-variance rule on a respaced cosine grid. For adjacent original training times it is the familiar DDPM update; skipping times defines an effective larger β. The deterministic panel uses DDIM with η = 0 on the same grid. Both obtain their prediction from the exact mixture score. Their finite-step output only approximates the target distribution, and their initial standard Gaussian approximates the nearly Gaussian terminal mixture.

DDIM keeps the noise direction and changes its scale

DDIM, by Jiaming Song, Meng and Ermon, constructs alternative sampling processes compatible with the same denoising training objective. At η = 0, there is no fresh noise in an update. From current time t to an earlier time s, use the prediction to estimate x₀, then combine that estimate with the same predicted noise direction at the next noise level:

Deterministic DDIM · arbitrary compatible times s < t
# One deterministic DDIM update from time t to s < t.
eps = denoiser(xt, t, condition)
x0_hat = (xt - sqrt(1 - a_t) * eps) / sqrt(a_t)
xs = sqrt(a_s) * x0_hat + sqrt(1 - a_s) * eps
# a_t and a_s are cumulative alpha_bar values, not alpha_t.

Deterministic DDIM maps each initial noise draw to a repeatable sample, with different draws providing variety. Its finite updates approximate continuous probability-flow dynamics, with discretization error that depends on the timestep grid and step count.

The stochastic and deterministic views meet

In continuous time, write the forward process as dx = f(x, t)dt + g(t)dW, with scalar diffusion coefficient g. Its reverse-time SDE uses drift f − g²sₜ while integrating from high t down to low t. The corresponding probability-flow ODE uses drift f − ½g²sₜ. With the exact score, matching initial distributions, and exact integration, these dynamics have the same time marginals. Their individual trajectories differ.

For the variance-preserving process, f(x, t) = −½β(t)x and g(t)² = β(t). Remember that dt is negative when using the reverse-time equation to sample. Mixing a reverse-time drift with a positive forward step is an easy way to make a sampler move the wrong way.

ProcedureRandomness after initializationMain trade-off
DDPM ancestralFresh noise at most updatesStochastic reverse transitions; coarse jumps distort them
DDIM · η = 0NoneSimple deterministic updates; large jumps accumulate error
Probability-flow ODE solverNoneSolver order and time grid trade evaluations for accuracy
Distilled / consistency modelDepends on inference procedureExtra training makes larger jumps useful
Where sampling error and sampling cost come from. Count network function evaluations, since a higher-order solver may use more than one evaluation per advertised step.

Learn the shortcut

You can improve a solver while leaving the denoiser fixed, or train a model that tolerates fewer evaluations. Progressive distillation trains a student to replace multiple teacher updates with fewer student updates. A consistency model instead learns a map that returns a common near-clean endpoint from different points on a probability-flow trajectory. It can be distilled from a diffusion teacher or trained through a consistency objective without that teacher.

Latent consistency models apply this idea in compressed image space and incorporate classifier-free guided teacher dynamics during distillation. The reported few-step performance belongs to the trained model and its inference procedure. Taking an ordinary checkpoint and simply changing its step count to one is a different experiment.

NoiseIntermediateNear-clean

Teacher: xₜ → xₛ → xδ

Student: Fθ(xₜ, t) ≈ xδ

Consistency: Fθ(xₜ, t) ≈ Fθ(xₛ, s)

  1. A teacher trajectory connects high noise to a small positive terminal noise δ.
  2. A student learns to predict the endpoint from a noisy point in one evaluation.
  3. Predictions from different times along the same trajectory should agree.
1 / 3
A consistency target identifies points along one trajectory with a shared endpoint. Distillation teaches the shortcut instead of merely deleting solver steps.
Check yourselfAn exact score is available. Why can eight DDIM steps still miss the target distribution?

Level 07/09Runtime

Make the image match the words

An unconditional denoiser can generate plausible images without knowing which image you want. Our prompt about the mug supplies a condition c. A text encoder converts its tokens into vectors that the denoiser can consult at every noise level. Public model families use encoders such as CLIP’s text transformer or T5, sometimes in combination. Their embeddings are learned representations of the words and context, rather than a symbolic scene specification.

In a cross-attention block, image features provide queries Q and text features provide keys K and values V. Softmax(QKᵀ/√dₕ)V produces a text-dependent update at each image position, where dₕ is a head’s key/query width. A feature near the developing mug can draw information from “coral” or “mug”; positions elsewhere can use the same prompt differently. The LLM inference dive develops the attention arithmetic in detail.

Cross-attention does not fix a permanent one-word-to-one-region correspondence. Attention weights vary across layers, heads and steps. Relationships such as “on a table” require the whole denoising network to coordinate features. Better text encoding can help, but it does not turn generation into guaranteed execution of a geometric program.

Text path · once“A coral mug…”Text encoderText tokens × embedding width
Image path · each evaluationNoisy latent + timeImage backbone + attention to textNoise / v / flow velocity prediction
The prompt is encoded once and reused. Every denoising evaluation combines the current noisy image, its time, and those text features.

Classifier-free guidance

Training with conditions gives a conditional prediction εc. To make that condition’s influence adjustable at inference time, randomly drop it on some training examples. The same network learns a reference prediction εu with an empty or null condition as well. At inference, evaluate both branches and combine them:

ε̂ = εu + w(εc − εu)

Here w = 0 uses the unconditional branch; w = 1 uses the ordinary conditional prediction; w > 1 extrapolates beyond it. Some papers and APIs write the scale as an extra weight on top of the conditional prediction, so their zero can correspond to our one. Check the equation before comparing scale numbers. Ho and Salimans introduced classifier-free guidance to obtain this fidelity-versus-diversity trade-off without a separate noisy-image classifier.

The same linear combination works on scores at a fixed noise level: ŝ = su + w(sc − su). At that time, it is the gradient of log[pₜ(x | c)ʷ pₜ(x)¹⁻ʷ], up to a normalization constant. For w > 1 it boosts directions associated with the condition relative to the unconditional distribution. Across the full process, these guided fields need not be the exact marginals of a simple clean-data distribution raised to a power.

Our conditional toy distribution contains the four right-hand clusters of the original ring. Its unconditional density still contains all eight. Both scores are analytic. At w = 1, the right-hand conditional sample should retain its four modes. Increase w and watch the conditional preference grow stronger while the final mass concentrates. Scrub backward to see when this preference starts separating particles.

Unconditional · w = 0

Guided · right-hand clusters

Exact mixture scores, 256 particles, cosine schedule. Circles show two standard deviations of the target clusters. Finite steps introduce sampling error.

Classifier-free guidance with exact conditional and unconditional scores. The left panel stays unconditional; the right extrapolates toward the four outlined right-hand clusters using the selected w.

High guidance can favour the most condition-distinctive modes at the expense of others. In images, strong extrapolation can also drive predicted clean values into extreme ranges and produce excessive saturation or harsh contrast. An error in either network branch is amplified too. The toy exposes concentration in a known distribution; real image artifacts also depend on the trained network, data scaling and sampler.

Replacing the unconditional branch with a negative-prompt condition makes the formula ε̂ = εnegative + w(εpositive − εnegative). That changes the reference direction. It helps explain both the usefulness and the limits of negative prompts: they reshape a numerical field using learned associations, with no explicit constraint solver checking the final picture.

Joint attention changes the conversation

SD3’s MMDiT uses separate weights for the image and text streams while allowing their tokens to interact through joint attention. Unlike a simple fixed-text cross-attention picture, both modalities participate in the transformer’s processing. Image and text tokens can contribute queries, keys and values to a combined attention computation. The architecture changes how the condition is used; the flow objective still determines what the output is trained to predict.

Two-branch guidance commonly concatenates conditional and unconditional copies into one batch of size two. The network shares weights, but it processes twice as many examples. Implementations can instead evaluate the branches sequentially to save peak activation memory. Distilled guidance and other model-specific methods can avoid the explicit two-branch runtime. The cost calculator below assumes ordinary two-branch guidance when its checkbox is selected.

Guidance extrapolates between two learned predictions. It buys stronger conditioning by changing the distribution and usually doing more network work.

Level 08/09System

Move the work into a smaller space

Our mug at 512 × 512 has 786,432 RGB values. Running a large network repeatedly at that spatial resolution is expensive. Much of that work concerns tiny pixel variations rather than the objects and relationships named in the prompt. Latent diffusion separates those jobs: learn a compressed image representation, model its distribution, then decode it back into pixels.

A A learned encoder and decoder trained with reconstruction and regularization objectives; its latent array stores a compressed representation of an image., or VAE, supplies the representation in many latent diffusion systems. Its encoder maps training images into latent arrays; its decoder maps latents back into images. A KL regularizer helps structure the latent space. During generation, the diffusion model starts with noise in that space, and the decoder runs after denoising finishes.

For a concrete factor-eight, four-channel configuration, 512 × 512 × 3 becomes 64 × 64 × 4: 16,384 latent scalars, 48 times fewer values. The number of spatial positions falls by 64. These ratios differ because the latent has four channels instead of three. This is an illustrative configuration used by public latent diffusion models, rather than a shape mandated by diffusion itself. Other autoencoders use different factors and channel counts.

Rombach and colleagues’ latent diffusion work studies the trade-off between perceptual compression and generative modeling cost. Compression can discard fine detail, so the decoder’s reconstruction quality imposes a limit of its own. A denoiser cannot recover information that the representation has no capacity to express, though the decoder can synthesize plausible detail.

RGB scalars
Latent · f = 8, C = 4
Image tokens

The grid previews the upper-left 8 × 8 latent cells. Borders group cells into patches; one patch becomes one token.

Attention pairs · N²
Latent storage · FP16
One dense score matrix · FP16
From pixels to latent cells to transformer patches. Change the image edge or patch edge and watch the exact scalar and token counts change.

A latent is a spatial array, not a tiny RGB image

Latent channels contain learned features. One channel does not simply mean red, another green and another blue. An implementation may also apply a model-specific scaling factor before diffusion so the latent distribution matches its training noise convention. The denoiser, scheduler and decoder must agree on that scaling. Getting the tensor dimensions right while getting its scale wrong still breaks the model.

For text-to-image generation, no clean source image goes through the encoder at runtime. You sample a noisy latent, denoise it and decode it. The encoder is used to make training latents and for image-conditioned tasks such as image-to-image. This distinction is useful when profiling: a text encoder and VAE decoder are usually outside the repeated denoising loop.

U-Net: local work at several scales

A U-Net repeatedly downsamples spatial features, processes lower-resolution representations and upsamples them again. Skip connections carry corresponding high-resolution features into the upsampling path. Convolutions do local mixing; attention blocks can add longer-range communication and text conditioning. Working at multiple scales lets a network coordinate a whole mug while preserving details near its rim.

The encoder-like side of this U shape is part of the denoiser. It is distinct from the VAE encoder that compresses the original image. The decoder-like side is likewise distinct from the VAE decoder that makes RGB pixels. They have similar spatial vocabulary because both use multiscale feature maps, but their inputs, training roles and runtime frequency differ.

DiT: turn latent patches into tokens

Peebles and Xie’s Diffusion Transformer, DiT, replaces the U-Net backbone with a transformer operating on latent patches. For patch edge p = 2, each 2 × 2 × 4 block becomes a vector of 16 values. A learned linear projection expands it to model width d. Positional information preserves the patch’s location, and timestep information conditions the transformer blocks.

At 512 pixels with factor-eight compression, the latent grid is 64 × 64. A 2 × 2 patch gives a 32 × 32 token grid, so N = 1024 image tokens. Transformer layers mix them, and an output projection predicts the required values for each patch. Unpatchifying places them back into the latent array. Patchifying reorganizes and projects values; it is not a second VAE.

U-Net

  1. 64 × 64 features
  2. 32 × 32 features
  3. 16 × 16 features
  4. 32 × 32 + skip features
  5. 64 × 64 + skip features

Schematic scales, with convolutions and optional attention at selected stages.

DiT

  1. 64 × 64 × 4 latent
  2. 1024 patches × 16 values
  3. 1024 tokens × width d
  4. Repeated transformer blocks
  5. Project and unpatchify

Example with 2 × 2 latent patches; time conditions the blocks.

Two backbones can predict the same target on the same latent tensor. The difference is how their internal features communicate.

Memory is spread across several different objects. Weights persist across all steps. The current latent is small. Intermediate feature maps, query/key/value tensors and MLP activations can be much larger. Two-branch guidance increases these activations because it increases the effective batch. Inference does not keep a full backpropagation tape, and it can release buffers as soon as later operations stop needing them.

Larger patches reduce token count at the cost of asking each token to represent a larger area. Smaller patches give the transformer finer spatial units and more computation. The original DiT study investigates precisely this relationship between patch size, model compute and generated quality. Choosing p is part of the trained architecture; a user cannot normally switch it on an existing checkpoint like changing the number of inference steps.

Check yourselfA 1024 × 1024 image uses factor-eight compression and 2 × 2 latent patches. How many image tokens enter a DiT?

Level 09/09Machine

Count the work inside one step

One denoising evaluation is now a familiar machine workload: projections, attention products and feed-forward matrix multiplications over image tokens. Unlike autoregressive text decoding, a typical image denoiser processes the whole spatial token array on every evaluation. All those positions change on the next evaluation, so there is no universally reusable text-style KV cache for their image keys and values.

Count a multiply-add as two floating-point operations. For N tokens of width d, dense Q, K and V projections cost about 6Nd² FLOPs, and the attention output projection adds 2Nd². With an MLP hidden width of 4d, its two main matrices add 16Nd². Together those projections and the MLP contribute about 24Nd² per transformer block.

The attention products QKᵀ and AV each cost about 2N²d across heads, giving 4N²d total. Splitting width d into more heads changes each head’s dimensions, but the leading product count still sums to this expression. This simplified block estimate leaves out normalization, activations, conditioning projections and other overhead:

Fblock ≈ 24Nd² + 4N²d
N = (H / (f · p)) × (W / (f · p))

Here H and W are pixel dimensions, f is the autoencoder’s spatial compression factor, and p is the latent patch edge. The calculator uses f = 8, d = 1024, an MLP ratio of four, and a hypothetical 24-block backbone.

RGB scalars
Latent · f = 8, C = 4
Image tokens

The grid previews the upper-left 8 × 8 latent cells. Borders group cells into patches; one patch becomes one token.

Attention pairs · N²
Latent storage · FP16
One dense score matrix · FP16

One transformer block · d = 1024 · MLP ratio 4

Projections + MLP
Attention products

Toy 24-block backbone; image self-attention only. Multiply-add counts as 2 FLOPs. Excludes text, normalization, embeddings and the VAE.

A dimension-based compute budget. Doubling both image edges gives four times as many tokens and sixteen times the attention-product work; the projection and MLP work grows fourfold.

Work grows differently in different parts of the block

At 512 pixels and p = 2, N is 1024. With d = 1024, projections and MLP cost about 25.77 GFLOPs per block, while the two attention products cost about 4.29 GFLOPs. At 1024 pixels, N is 4096: those terms become about 103.08 and 68.72 GFLOPs. Attention is not automatically the largest term at every resolution.

“Quadratic in resolution” needs a unit. Dense attention is quadratic in token count, which is proportional to pixel area when compression and patch size are fixed. If you double both image edges, pixel area and N grow fourfold, and N² grows sixteenfold. In terms of the edge of a square image, that term grows as the fourth power. The rest of the block has a large term linear in token count.

Network evaluations multiply this work again. Thirty evaluations of a 24-block toy backbone with two-branch guidance cost about 43.29 TFLOPs at the 512-pixel settings. Forty evaluations do roughly a third more denoiser arithmetic. A more sophisticated solver might use two evaluations for one step, so counting calls to the denoiser is often clearer than reading the step counter in a UI.

The current latent itself is a surprisingly small part of memory. Our 64 × 64 × 4 example occupies 32 KiB in FP16. A hypothetical one-billion-parameter network needs about 2 GB just for its FP16 weights, before temporary buffers or text encoders. The runtime reuses those same weights at each denoising evaluation. Reducing the step count saves repeated work, but it does not divide the resident weight storage by the number of steps.

Batching several requested images can improve hardware utilization by putting more matrix work into each invocation. It also multiplies activation storage and keeps the image trajectories independent: one mug’s current latent is not used as another mug’s history. Two-branch guidance is an additional batch factor within each request. Distinguishing request batch size from guidance branches makes memory estimates and throughput measurements much easier to interpret.

Memory can be the limit before arithmetic is

A single dense attention score matrix has N² entries per head and per batch element. At N = 4096, FP16 storage is 32 MiB for one matrix. Sixteen heads and a two-element guidance batch would make that particular materialization 1 GiB. This is separate from weights and the other live activations. The calculator reports one matrix so you can apply the head and batch multipliers explicitly.

Memory-efficient attention computes softmax in tiles and accumulates the output without storing that full matrix. It changes memory traffic and temporary storage dramatically while retaining the leading dense-attention arithmetic. The LLM inference dive explains tiled attention and GPU matrix multiplication. The same ideas apply when the tokens represent latent patches instead of words.

At modest resolution, repeated weight reads, kernel launches, normalization and device utilization also affect latency. At high resolution, feature and attention work grows. Peak advertised GPU FLOPs are therefore not a stopwatch conversion factor. To estimate actual latency, measure the selected model, shapes, precision and kernels on the target device.

ChangeBudget term affectedWhat to check
Fewer evaluationsRepeated denoiser workSolver error or distillation quality
Smaller latent / larger patchesN and N²Representation quality and architecture compatibility
Tiled attentionMemory traffic and score storageSupported shapes, kernels and precision
Quantized weightsWeight storage and possibly computeKernel support and quality across noise levels
Feature cachingSelected repeated computationsError from reusing stale image features
Guidance distillationExplicit conditional/reference branchesGuidance range supported by the student
Speedups act on different factors in the budget. Their compatibility depends on the trained model and runtime.

Quantization stores weights or activations with fewer bits. The quantization dive explains scales, rounding and the difference between a smaller file and faster kernels. For a diffusion model, activation distributions can vary strongly with noise level; a scheme that works near the clean endpoint can still damage early denoising predictions. Errors also feed into the next evaluation, so final image quality must be checked across the whole sampling trajectory.

Caching needs a similar distinction. Fixed text embeddings are safe to reuse. Image features evolve, and reusing them across steps generally approximates the computation. A runtime may find that some blocks change slowly enough to skip selected updates, but that is a measured speed-quality trade-off. The usual autoregressive argument that old token keys remain valid does not apply automatically to a denoising array.

Back to the mug

The prompt became text embeddings. The seed became a Gaussian latent. A schedule told the denoiser which noise level it was seeing. The model predicted noise, v, or a flow velocity according to its training contract. A sampler combined that prediction with its update rule; guidance changed the field using a second condition. Repeating the process produced a latent that a decoder could turn into pixels.

At the bottom, our mug is matrix products over a spatial array, paid for again at every evaluation. The learned field provides the knowledge; the solver determines how often to ask for it; compression determines the size of the question. Those are three separate places to improve quality or save work, and three separate places to look when a generation goes wrong.

Generation cost is repeated network work over spatial tokens. Step count multiplies it; resolution changes both its linear and quadratic terms.

End of Diffusion · 9 levels, Idea to Machine

Where to next

Browse all 34 divesAll topics