Skip to content

Samplers and schedules

A trained diffusion model only tells us, for a noisy image, what it thinks the clean image is. It’s the sampler that turns that into a picture: it starts from pure noise and takes a series of steps down the noise levels. In notebook 01 we wrote DDIM, Euler, DDPM and Heun by hand and talked about how each of them removes noise. Here we take the flower model we trained in notebook 02. We compare seven built-in samplers at a few step counts, all from the same starting noise, and time each one.

Five data points on a circle and the paths three solvers take to them from the same starting noise: a fine-step reference path, Euler in three steps, which stops short between the points, and Heun in three steps, which lands on one.datax_Treference pathEuler, 3 callsHeun, 6 calls

That’s the whole story in two dimensions. From the same starting noise, Euler’s three steps stop short of the data, and Heun’s, which ask the model twice each, land on it.

The model loads from the Hub, where notebook 02’s run is saved as dewml/flowers-dit-35m.

This time we only need Dew and its cuda13 extra.

%pip install -q "dewml[cuda13] @ git+https://github.com/AshishKumar4/dew@fcc2bf6f5141d7971f9ad7886aa8592a015e2fdc"
install log
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
  Building wheel for dewml (pyproject.toml) ... done

STEP_COUNTS are the numbers of noise levels each sampler uses, and SAMPLES is how many images each run draws.

IMAGE_SIZE = 128
CHECKPOINT = "dewml/flowers-dit-35m"
STEP_COUNTS = [10, 20, 40]
SAMPLES = 4
SEED = 0
import time
import dew # before JAX's backend opens, so its numerical settings apply
import jax
import jax.numpy as jnp
import numpy as np
from dew import figures
from dew.artifacts import uint8_pixels
print(jax.devices())
[CudaDevice(id=0)]

Remember from notebook 02: a checkpoint holds arrays, not code. Next to the arrays, though, each checkpoint records how to rebuild the model, its noise process and its sampler, and dew.pipeline reads that record back with the weights. So we don’t repeat notebook 02’s settings here. Of the weights it reads the averaged (EMA) copy the run kept, the one notebook 02 sampled from.

task = dew.pipeline(CHECKPOINT)
model, process, params = task.model, task.process, task.variables
print(f"{sum(x.size for x in jax.tree.leaves(params)) / 1e6:.1f}M parameters")
stderr
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
WARNING:huggingface_hub.utils._http:Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
35.5M parameters

To compare samplers fairly, everything except the sampler has to stay fixed. So we draw the starting noise xTx_T once and hand the same array to every run. Its spread is about 80, the highest noise level of the EDM schedule. The ancestral samplers (DDPM and Euler ancestral) also add fresh noise at every step; sample derives that noise from the key we pass, and we pass the same key every time.

x_T = process.noise(jax.random.key(SEED), (SAMPLES, IMAGE_SIZE, IMAGE_SIZE, 3))
print("x_T spread:", float(jnp.std(x_T)))
x_T spread: 79.90058898925781

If any of these feel unfamiliar, notebook 01 explains DDIM, Euler, DDPM and Heun in a lot more detail.

  • DDPM follows the reverse Markov chain of the original DDPM paper by Ho et al. (2020) and adds fresh noise at every step. It needs many small steps.
  • DDIM, from the DDIM paper by Song et al. (2020), is the deterministic version: jump to the next noise level along the direction the model points to.
  • Euler takes one Euler step of the probability flow ODE per noise level. On this schedule it is the same update as DDIM, just written differently; this is the algebraic equivalence I left as an exercise in notebook 01.
  • Euler ancestral takes an Euler step to a slightly lower level and then adds back some fresh noise, so it solves the reverse SDE instead of the ODE.
  • Heun takes an Euler step, asks the model again at the end of it, and averages the two slopes. That costs two model calls per step.
  • RK4 is the classical fourth-order Runge-Kutta method: four model calls per step.
  • MultiStepDPM reuses the model’s answers from the last three steps to correct the current one, at one model call per step.

Each is a small value in dew.sampling. As in notebook 02, process.denoiser(model, params, {}) wraps the model and its weights into the function a sampler calls. Then sample(denoise, x_T, steps, solver=..., key=...) runs any of the samplers over it in one loop.

from dew.sampling import DDIM, DDPM, RK4, Euler, EulerAncestral, Heun, MultiStepDPM, sample
SOLVERS = {
"DDPM": DDPM(),
"DDIM": DDIM(),
"Euler": Euler(),
"Euler ancestral": EulerAncestral(),
"Heun": Heun(),
"RK4": RK4(),
"MultiStepDPM": MultiStepDPM(),
}

To time the samplers fairly, we compile each one once with jax.jit and time only a second call, otherwise we would mostly be timing XLA’s compiler rather than the sampler. The step count and the solver are static arguments, so every combination compiles its own loop. The weights are a normal argument, which keeps them out of the compiled program.

import functools
@functools.partial(jax.jit, static_argnames=("steps", "solver"))
def run(params, x_T, key, steps, solver):
denoise = process.denoiser(model, params, {})
return sample(denoise, x_T, steps, solver=solver, key=key)
images, seconds = {}, {}
for steps in STEP_COUNTS:
for name, solver in SOLVERS.items():
run(params, x_T, jax.random.key(SEED), steps, solver).block_until_ready()
start = time.perf_counter()
out = run(params, x_T, jax.random.key(SEED), steps, solver).block_until_ready()
seconds[name, steps] = time.perf_counter() - start
images[name, steps] = np.asarray(out)

The grid flips through the three step counts. Each row is a sampler and each column one starting noise, so images in the same column started from the same point.

frames = [uint8_pixels(np.concatenate([images[name, steps] for name in SOLVERS]))
for steps in STEP_COUNTS]
figures.images(frames, columns=SAMPLES, size=72, rows=list(SOLVERS),
captions=[f"{steps} steps" for steps in STEP_COUNTS])
28 images at each of 3 pointsDDPMDDIMEulerEuler ancestralHeunRK4MultiStepDPM10 steps20 steps40 steps

At 10 steps, DDPM and Euler ancestral leave soft, almost featureless blobs, while the deterministic samplers already draw the same four flowers. Heun and RK4 are sharper than Euler at that step count: their petals have edges where Euler’s are still blurred. By 40 steps the deterministic samplers all look alike. The ancestral samplers have taken different paths from that shared starting noise, and at each step count they land somewhere a little different.

The timings broadly follow the number of model calls. At 40 steps RK4 takes 412 ms against Euler’s 117 ms, three and a half times as long, and Heun takes 230 ms, about twice as long. At 10 steps the corresponding times are 106, 38 and 72 ms. Each entry is one timed call after a warm-up call, not a repeated benchmark.

print(f"{'sampler':<16}" + "".join(f"{steps:>8} steps" for steps in STEP_COUNTS))
for name in SOLVERS:
print(f"{name:<16}" + "".join(f"{seconds[name, steps] * 1000:>11.0f} ms" for steps in STEP_COUNTS))
sampler               10 steps      20 steps      40 steps
DDPM                     39 ms         75 ms        140 ms
DDIM                     39 ms         66 ms        125 ms
Euler                    38 ms         73 ms        117 ms
Euler ancestral          39 ms         75 ms        138 ms
Heun                     72 ms        124 ms        230 ms
RK4                     106 ms        204 ms        412 ms
MultiStepDPM             39 ms         77 ms        142 ms

A fast sampler that lands somewhere else isn’t a bargain, though. So let’s put each deterministic sampler’s time next to how far it lands from the answer. For the answer we take a 200-step Heun solve from the same noise. The ancestral samplers sit this one out: they add noise on purpose, so they aren’t aiming for that answer.

fine = np.asarray(run(params, x_T, jax.random.key(SEED), 200, Heun()))
deterministic = ["Euler", "Heun", "RK4", "MultiStepDPM"]
for name in deterministic:
print(f"{name:<14}" + " ".join(
f"{steps} steps: {seconds[name, steps] * 1000:5.0f} ms, "
f"{np.mean(np.abs(images[name, steps] - fine)):.4f}" for steps in STEP_COUNTS))
figures.points({name: [(seconds[name, steps] * 1000,
np.mean(np.abs(images[name, steps] - fine)))
for steps in STEP_COUNTS] for name in deterministic},
x_label="milliseconds per run",
y_label="distance from the 200-step solve", log_x=True, log_y=True)
Euler         10 steps:    38 ms, 0.1155  20 steps:    73 ms, 0.0674  40 steps:   117 ms, 0.0359
Heun          10 steps:    72 ms, 0.0429  20 steps:   124 ms, 0.0149  40 steps:   230 ms, 0.0071
RK4           10 steps:   106 ms, 0.0115  20 steps:   204 ms, 0.0118  40 steps:   412 ms, 0.0047
MultiStepDPM  10 steps:    39 ms, 0.0916  20 steps:    77 ms, 0.0365  40 steps:   142 ms, 0.0110
distance from the 200-step solve against milliseconds per run0.0050.010.020.050.150100200milliseconds per rundistance from the 200-step solveEulerHeunRK4MultiStepDPM

Heun at 10 steps lands about as close as Euler at 40, in less time: 0.043 in 72 ms against 0.036 in 117 ms. MultiStepDPM costs about what Euler costs and lands three times closer at 40 steps, 0.011 against 0.036. RK4 is closest of all at 10 steps, 0.012 in 106 ms, though doubling its steps doesn’t help. So a sampler that asks the model more than once per step, or reuses its earlier answers, buys a lot more accuracy than the extra milliseconds would suggest.

The deterministic samplers all solve the same ODE, so with enough steps they should land on nearly the same images. Below is the mean absolute pixel difference between each sampler’s images and Euler’s at 40 steps, on the [-1, 1] scale. The ancestral samplers add their own noise along the way, so they end up at different flowers.

The last line compares the two ancestral samplers with each other: with the same key they give almost the same images, so on this schedule their updates nearly coincide. That is expected: on this schedule, where the signal rate is 1, Dew’s DDPM step (with its default variance="small") and Euler ancestral both work out to x^0+(σs2/σt) ϵ^+σupz\hat{x}_0 + (\sigma_s^2 / \sigma_t)\,\hat{\epsilon} + \sigma_\text{up} z, so in exact arithmetic they are the same update. It is DDIM with η=1\eta = 1 (Song, Meng and Ermon, 2021), one particular discretization of the reverse diffusion SDE.

reference = images["Euler", STEP_COUNTS[-1]]
for name in SOLVERS:
difference = np.mean(np.abs(images[name, STEP_COUNTS[-1]] - reference))
print(f"{name:<16} {difference:.4f}")
print("DDPM against Euler ancestral:", np.mean(np.abs(images["DDPM", STEP_COUNTS[-1]] - images["Euler ancestral", STEP_COUNTS[-1]])))
DDPM             0.3803
DDIM             0.0000
Euler            0.0000
Euler ancestral  0.3804
Heun             0.0381
RK4              0.0372
MultiStepDPM     0.0269
DDPM against Euler ancestral: 0.00064495887

Presets pair a training schedule with a sampling schedule

Section titled “Presets pair a training schedule with a sampling schedule”

A sampler uses whatever noise levels the process gives it, and those have to match the model: the sampling rates, the model’s time and the prediction transform all have to be the ones it was trained with. That is why Dew keeps both in one Process. EDM trains on log-normal noise levels and samples on the Karras grid. The cosine preset trains and samples one discrete schedule with v-prediction, the parameterization from the paper “Progressive Distillation for Fast Sampling of Diffusion Models” by Salimans and Ho (2022). The flow preset is rectified flow, from the paper “Flow Straight and Fast” by Liu et al. (2022), on both sides.

Sampling a model with different rates, time or prediction transform than it trained with gives the wrong result. The noise levels it trains on can differ, though: EDM and Karras train on different ones and sample the same way.

Here are the noise levels each preset’s sampler steps through on a grid of 21 levels, leaving out the two ends, where the flow preset reaches pure noise and pure signal. The presets write their noise levels differently, so the chart puts them on one axis as the signal-to-noise ratio α2/σ2\alpha^2 / \sigma^2, on a log scale.

from dew.diffusion.presets import EDM, Cosine, Flow, Karras
for name, preset in (("edm", EDM(regime="pixel")), ("karras", Karras()),
("cosine", Cosine()), ("flow", Flow())):
built = preset()
print(f"{name:<7} train: {type(built.schedule).__name__:<24} "
f"sample: {type(built.sampler_schedule).__name__:<24} "
f"predicts: {type(built.prediction).__name__}")
grids = {}
for name, preset in (("edm, karras", EDM(regime="pixel")), ("cosine", Cosine()),
("flow", Flow())):
built = preset()
# The flow preset's two ends are pure noise and pure signal, infinitely far
# apart on this scale.
alphas, sigmas = built.sampler_schedule.rates(built.times(21)[1:-1])
grids[name] = np.log10(np.asarray(alphas) ** 2 / np.asarray(sigmas) ** 2)
figures.curve(grids, x_label="sampling step", y_label="log₁₀ α² / σ²")
edm     train: EDMNoiseScheduler        sample: KarrasVENoiseScheduler   predicts: KarrasPredictionTransform
karras  train: KarrasVENoiseScheduler   sample: KarrasVENoiseScheduler   predicts: KarrasPredictionTransform
cosine  train: CosineNoiseScheduler     sample: CosineNoiseScheduler     predicts: VPredictionTransform
flow    train: FlowMatchingScheduler    sample: FlowMatchingScheduler    predicts: FlowMatchPredictionTransform
log₁₀ α² / σ² against sampling step-202405101520sampling steplog₁₀ α² / σ²edm, karras 4.41cosine 2.08flow 2.56

EDM’s Karras grid covers far more ground than the other two: from a signal-to-noise ratio of about 10−3.610^{-3.6} to 104.410^{4.4}, eight orders of magnitude, where cosine covers about four and flow five. Cosine and flow pass a ratio of 1 halfway through their steps; EDM is still at about 10−0.810^{-0.8} there, so it packs its steps more tightly at the noisy end.

Notebook 03 adds a text caption and classifier-free guidance on top of the very same sample call.