Skip to content

dew.sampling.solvers

Diffusion solvers, each taking one reverse step from t to t_next with the model’s prediction at t.

A solver is a value, and a pytree: its settings are static, so they shape the compiled walk, and only a number a walk varies by request (DPMSolverSDE’s seed) is a leaf. Whatever it needs between steps is kept in its state, which init builds and step carries through sample’s scan. The signal and noise rates come from process. A solver that needs another model evaluation (Heun’s corrector, RK4’s stages, KDPM2’s midpoint) calls denoise. A solver that integrates dx / dsigma = eps refuses a schedule whose alpha is not one.

The solvers named after Diffusers 0.34.0 schedulers reproduce their arithmetic. Before the compiled scan, init checks each algorithm’s endpoint domain on the concrete grid; finite endpoint limits are tested, and endpoints where the update is undefined raise an error instead of substituting another update. A multistep update, which divides by the gaps between its history’s levels, refuses a grid with two equal levels before its last interval.

NameSummary
DDIMThe DDIM update, where eta sets the stochasticity: 0 is deterministic, 1 is DDPM-like.
DDPMExact ancestral sampler for the reverse diffusion SDE.
DEISDEIS (Zhang and Chen 2023, arXiv 2204.13902) in its log-rho multistep form, Diffusers 0.34.0’s DEISMultistepScheduler: the exponential integrator of eps with the polynomial-in-log(rho) interpolation of the last outputs, rho = sigma / alpha, integrated in closed form over the step.
KDPM2k-diffusion’s DPM-Solver-2 (sample_dpm_2), the update of Diffusers 0.34.0’s KDPM2DiscreteScheduler, and with ancestral its sample_dpm_2_ancestral and KDPM2AncestralDiscreteScheduler.
LMSLinear multistep over dx/dsigma = (x - x_0) / sigma, k-diffusion’s sample_lms and Diffusers 0.34.0’s LMSDiscreteScheduler: the last order derivatives interpolated by the Lagrange polynomial through their sigmas and integrated over the step, in closed form where Diffusers quadratures; the order grows with the history.
PNDMPNDM as Diffusers 0.34.0’s PNDMScheduler: Adams-Bashforth over eps with DDIM as the transfer.
RK4Classical fourth-order Runge-Kutta over dx/dsigma = eps on a variance-exploding schedule.
TCDTrajectory consistency sampling, as Diffusers 0.34.0’s TCDScheduler.
ConsistencyMultistep consistency sampling, the update of Diffusers 0.34.0’s LCMScheduler.
DPMSolverMultistepDPM-Solver and DPM-Solver++ as multistep integrators in lambda = log(alpha) - log(sigma).
DPMSolverSDEDiffusers 0.34.0’s DPMSolverSDEScheduler, k-diffusion’s sample_dpmpp_sde midpoint solver over a Brownian tree.
DPMSolverSinglestepDiffusers 0.34.0’s grouped DPM-Solver updates from each group’s anchor.
EulerThe DDIM update written as an Euler step of the probability flow ODE.
EulerAncestralEuler with k-diffusion’s ancestral noise injection (get_ancestral_step, eta 1).
HeunHeun’s second-order method: an Euler step, the derivative re-evaluated at its end, and their average.
MultiStepDPMA third order multistep integrator of dx/dsigma = eps on a variance exploding schedule, from finite differences of the last three eps: FlaxDiff’s MultiStepDPM, which despite the name is not DPM-Solver.
RowKeysOne typed PRNG key per row of a batch, keys [B], for a solver whose draws are row by row (RowSolver): a row’s draws are its own key’s, whatever rows sample beside it, as they are sampled alone, while the model still runs once over the batch.
RowSolverA solver that draws each row by its own key (RowKeys): sample hands step_rows the step’s row keys where it hands step one key.
SolverThe interface for one diffusion solver step and the state it keeps between steps.
UniPCUniPC, a unified predictor and corrector in lambda, as Diffusers 0.34.0’s UniPCMultistepScheduler.

dataclass source

class DDIM(eta: float = struct.field(pytree_node=False, default=0.0))

The DDIM update, where eta sets the stochasticity: 0 is deterministic, 1 is DDPM-like.

DDIM is from Song et al. (2021). Diffusers 0.34.0’s DDIMScheduler limits the clean prediction under clip_sample or thresholding and keeps the model’s own output as its epsilon, so the direction term uses the unlimited one. That pairing comes from SourceLimitedPrediction, in the process’s conversion.

def init(x, times, process, *, key)
def step(x, t, t_next, denoised, eps, state, key, process, denoise)

dataclass source

class DDPM(
variance: Literal['small', 'large'] = struct.field(pytree_node=False, default='small'),
)

Exact ancestral sampler for the reverse diffusion SDE.

One step draws from the forward posterior q(x_s | x_t, x_0) for x_t = alpha_t x_0 + sigma_t eps, written in signal and noise rates so it holds for any schedule and any step stride. The posterior mean is alpha_s x_0 + alpha_t sigma_s^2 / (alpha_s sigma_t) eps and its variance is sigma_s^2 (1 - alpha_t^2 sigma_s^2 / (alpha_s^2 sigma_t^2)).

variance selects which of Diffusers 0.34.0’s fixed DDPMScheduler posterior variances the draw uses. "small" is the posterior’s own variance (posterior_std). "large" is the forward step’s own noise (transition_variance), the wider choice from Glide, which Diffusers writes as the beta 1 - alpha_t^2 / alpha_s^2 of a variance-preserving table. Both are written in rates, so they are defined on any schedule.

Neither choice adds noise on the step whose own time is the schedule’s zero, where x_t is already the least noisy state the schedule holds; the source skips its draw at that time the same way. Elsewhere, when the target’s alpha is one on a variance-preserving schedule, the wide noise has standard deviation sigma_t, the square root of the source’s own current_beta_t.

def init(x, times, process, *, key)
def step(x, t, t_next, denoised, eps, state, key, process, denoise)

dataclass source

class DEIS(
order: int = struct.field(pytree_node=False, default=2),
lower_order_final: bool = struct.field(pytree_node=False, default=True),
)

DEIS (Zhang and Chen 2023, arXiv 2204.13902) in its log-rho multistep form, Diffusers 0.34.0’s DEISMultistepScheduler: the exponential integrator of eps with the polynomial-in-log(rho) interpolation of the last outputs, rho = sigma / alpha, integrated in closed form over the step. The first order is DPM-Solver’s. Orders grow with the history; lower_order_final is the same short-walk taper as DPMSolverMultistep’s. At sigma=0 the integrated log-rho basis retains its history terms. An alpha=0 source contributes a node at infinite rho; that node’s weight vanishes in subsequent finite-interval integrals.

def init(x, times, process, *, key)
def step(x, t, t_next, denoised, eps, state, key, process, denoise)

dataclass source

class KDPM2(ancestral: bool = struct.field(pytree_node=False, default=False))

k-diffusion’s DPM-Solver-2 (sample_dpm_2), the update of Diffusers 0.34.0’s KDPM2DiscreteScheduler, and with ancestral its sample_dpm_2_ancestral and KDPM2AncestralDiscreteScheduler.

It takes an Euler step to the geometric midpoint of sigma_t and the target level, evaluates the model there, and steps from x with that midpoint derivative. The target is sigma_s, or under ancestral the sigma_down of k-diffusion’s ancestral step with sigma_up of fresh noise added after. The midpoint’s time comes from the schedule’s t_of_sigma, so this integrates a GeneralizedNoiseScheduler.

def init(x, times, process, *, key)
def step(x, t, t_next, denoised, eps, state, key, process, denoise)

dataclass source

class LMS(order: int = struct.field(pytree_node=False, default=4))

Linear multistep over dx/dsigma = (x - x_0) / sigma, k-diffusion’s sample_lms and Diffusers 0.34.0’s LMSDiscreteScheduler: the last order derivatives interpolated by the Lagrange polynomial through their sigmas and integrated over the step, in closed form where Diffusers quadratures; the order grows with the history. Integrates a GeneralizedNoiseScheduler.

def init(x, times, process, *, key)
def step(x, t, t_next, denoised, eps, state, key, process, denoise)

dataclass source

class PNDM(skip_prk_steps: bool = struct.field(pytree_node=False, default=False))

PNDM as Diffusers 0.34.0’s PNDMScheduler: Adams-Bashforth over eps with DDIM as the transfer.

PNDM is from Liu et al. (2022, arXiv 2202.09778). It is fourth order once four outputs are available. The warmup is the paper’s, three pseudo Runge-Kutta steps of four model evaluations each (the stages at the interval’s midpoint and end), which seed the history with each step’s first eps; under skip_prk_steps, the PLMS form Stable Diffusion runs, the first step is a predictor-corrector pair (an Euler step, eps re-read at its end, the step retaken with the mean) and the orders grow from there. The schedule sets the transfer stride: ordinary native grids use their adjacent interval, while published integer grids retain their fixed training stride. Transfers cannot start at alpha = 0.

def init(x, times, process, *, key)
def step(x, t, t_next, denoised, eps, state, key, process, denoise)

dataclass source

class RK4()

Classical fourth-order Runge-Kutta over dx/dsigma = eps on a variance-exploding schedule.

The half-step stages evaluate the model at the time the schedule maps their sigma back to. This is FlaxDiff’s RK4Sampler.

def init(x, times, process, *, key)
def step(x, t, t_next, denoised, eps, state, key, process, denoise)

dataclass source

class TCD(eta: float = struct.field(pytree_node=False, default=0.3))

Trajectory consistency sampling, as Diffusers 0.34.0’s TCDScheduler.

This is from Zheng et al. (2024, arXiv 2402.19159). The deterministic DDIM step lands at (1 - eta) t_next, and the forward process noises it up to t_next with fresh noise; this is the paper’s gamma-sampling with gamma = eta. At eta 0 it is DDIM; at eta 1 the step goes through the clean end of the schedule. A tabulated schedule reads the intermediate time at its truncated index, as the reference floors it. The last step of a walk lands on the grid’s end itself and takes no noise.

def init(x, times, process, *, key)
def step(x, t, t_next, denoised, eps, state, key, process, denoise)

dataclass source

class Consistency()

Multistep consistency sampling, the update of Diffusers 0.34.0’s LCMScheduler.

This is Algorithm 1 of Song et al. (2023). The clean prediction is noised again to the next level with fresh noise, x_s = alpha_s x_0 + sigma_s z, and the last step keeps x_0 as it is. A latent consistency model’s x_0 is the consistency function’s output, which ConsistencyBoundary reads out of the model’s prediction.

def init(x, times, process, *, key)
def step(x, t, t_next, denoised, eps, state, key, process, denoise)

dataclass source

class DPMSolverMultistep(
order: int = struct.field(pytree_node=False, default=2),
algorithm: Algorithm = struct.field(pytree_node=False, default='dpmsolver++'),
solver_type: Literal['midpoint', 'heun'] = struct.field(pytree_node=False, default='midpoint'),
lower_order_final: bool = struct.field(pytree_node=False, default=True),
euler_at_final: bool = struct.field(pytree_node=False, default=False),
)

DPM-Solver and DPM-Solver++ as multistep integrators in lambda = log(alpha) - log(sigma).

The papers are Lu et al. (2022, arXiv 2206.00927) and arXiv 2211.01095. This covers the four algorithms, three orders and two second-order forms of Diffusers 0.34.0’s DPMSolverMultistepScheduler, with its defaults.

dpmsolver++ and sde-dpmsolver++ integrate the clean prediction, the other two eps; the sde- forms add the noise term of the SDE solver. With h = lambda_t - lambda_s0 over the step and D0, D1, D2 the finite differences of the last outputs in lambda, the deterministic dpmsolver++ step is

x_t = (sigma_t / sigma_s0) x - alpha_t (e^-h - 1) D0 + c D1 + c_2 D2

with each algorithm’s coefficients written as Diffusers writes them. The first step has no history and is first order, the second at most second. lower_order_final is Diffusers’ rule verbatim, which acts only in a walk under 15 steps: first order on the last step and at most second on the one before. euler_at_final makes the last step first order whatever the length. A zero target forces the clean limit for deterministic and ++ updates. The non-++ SDE first-order limit also retains alpha_tsigma_s/alpha_s(noise-eps); it requires a finite source alpha and a final order reduction. That endpoint is an equation-limit extension: Diffusers refuses a literal zero-terminal non-++ config. EDM uses the ++ algorithms over its existing process.

DPM-Solver++ 2M with no order taper is order=2, algorithm=“dpmsolver++”, solver_type=“midpoint”, lower_order_final=False, euler_at_final=False.

def init(x, times, process, *, key)
def step(x, t, t_next, denoised, eps, state, key, process, denoise)

dataclass source

class DPMSolverSDE(
depth: int = struct.field(pytree_node=False, default=MAX_BROWNIAN_DEPTH),
seed: int | None = None,
)

Diffusers 0.34.0’s DPMSolverSDEScheduler, k-diffusion’s sample_dpmpp_sde midpoint solver over a Brownian tree.

Each interval takes two ancestral first-order steps from its own start: one to the geometric midpoint of sigma_t and sigma_s, where the model is evaluated, and one to sigma_s with that midpoint’s clean prediction. Both steps go down to k-diffusion’s sigma_down and add sigma_up of noise, and both draw that noise from one Brownian path over the trajectory’s sigma interval: the first over [sigma_t, sigma_mid] and the second over [sigma_t, sigma_s], so the two are correlated exactly as nested increments of one path. The source’s sampler transforms sigma with the identity even though its own steps integrate -log(sigma), so the interval widths are sigma differences.

depth resolves the root interval to (sigma_max - sigma_min) / 2**depth: on a published VP table’s span of about 14.6 the default reaches 8.7e-7, at or inside the reference tree’s own 1e-6 tolerance, and it is also where a float32 position runs out of mantissa, so no deeper descent tells two sigmas apart. A zero-sigma target has no ancestral step and lands on the clean prediction.

The root interval is the schedule’s positive sigma domain, not the grid’s extremes, as the source builds its tree over every positive sigma it prepared, so a walk over a suffix of the grid keeps the whole grid’s path.

seed is the source’s noise_sampler_seed: the path is then fixed by the checkpoint, independent of the walk’s key. A Torch seed names no JAX stream, so it seeds this bridge rather than reproducing the reference tree. It is the solver’s one pytree leaf, so a compiled walk takes another seed as data; the rest is static.

def init(x, times, process, *, key)
def step(x, t, t_next, denoised, eps, state, key, process, denoise)

dataclass source

class DPMSolverSinglestep(
order: int = struct.field(pytree_node=False, default=2),
algorithm: Literal['dpmsolver++', 'dpmsolver', 'sde-dpmsolver++'] = struct.field(pytree_node=False, default='dpmsolver++'),
solver_type: Literal['midpoint', 'heun'] = struct.field(pytree_node=False, default='midpoint'),
lower_order_final: bool = struct.field(pytree_node=False, default=False),
)

Diffusers 0.34.0’s grouped DPM-Solver updates from each group’s anchor.

A group of k model evaluations completes one k-th order update. Orders repeat [1, 2] or [1, 2, 3]; an incomplete final group uses lower order, as set_timesteps does in the reference. lower_order_final also lowers the final complete group, and a zero-sigma target forces final order 1. Source-domain checks use that effective order list.

At alpha=0, clean-prediction midpoint and deterministic Heun groups have finite limits. Noise-prediction groups above order 1 and third-order SDE Heun groups diverge; initialization rejects those source/grid pairs.

def order_list(steps: int) -> list[int]

Reference groups, completing an uneven final group at lower order.

def init(x, times, process, *, key)
def step(x, t, t_next, denoised, eps, state, key, process, denoise)

dataclass source

class Euler()

The DDIM update written as an Euler step of the probability flow ODE. On a variance exploding schedule it is dx/dsigma = eps.

def init(x, times, process, *, key)
def step(x, t, t_next, denoised, eps, state, key, process, denoise)

dataclass source

class EulerAncestral()

Euler with k-diffusion’s ancestral noise injection (get_ancestral_step, eta 1).

Each step goes down to sigma_down, then adds fresh noise of standard deviation sigma_up to bring the marginal back to sigma_s. It integrates a GeneralizedNoiseScheduler.

def init(x, times, process, *, key)
def step(x, t, t_next, denoised, eps, state, key, process, denoise)

dataclass source

class Heun(
s_churn: float = struct.field(pytree_node=False, default=0.0),
s_tmin: float = struct.field(pytree_node=False, default=0.0),
s_tmax: float = struct.field(pytree_node=False, default=float('inf')),
s_noise: float = struct.field(pytree_node=False, default=1.0),
)

Heun’s second-order method: an Euler step, the derivative re-evaluated at its end, and their average.

This is Algorithm 2 of Karras et al. (2022).

s_churn, s_tmin, s_tmax and s_noise are the algorithm’s stochasticity. At a sigma in [s_tmin, s_tmax] each step first raises the noise level to sigma (1 + gamma), gamma = min(s_churn / N, sqrt(2) - 1) for an N-interval walk, by adding fresh noise of standard deviation s_noise sqrt(sigma_hat^2 - sigma^2), and then takes the Heun step from there. The churn moves sigma directly, so it needs a variance-exploding schedule, and a raised level past the schedule’s top has no model time, so a sampling run that would churn there is refused. With churn, the step evaluates the model at the raised level, and the evaluation sample made at the grid point is unused, so the compiler removes it.

Diffusers 0.34.0’s HeunDiscreteScheduler limits the clean prediction of both stages under clip_sample; that limit belongs to the process’s conversion, SourceLimitedPrediction, so both evaluations here read the limited prediction without the solver knowing about it.

def init(x, times, process, *, key)
def step(x, t, t_next, denoised, eps, state, key, process, denoise)

dataclass source

class MultiStepDPM()

A third order multistep integrator of dx/dsigma = eps on a variance exploding schedule, from finite differences of the last three eps: FlaxDiff’s MultiStepDPM, which despite the name is not DPM-Solver.

def init(x, times, process, *, key)
def step(x, t, t_next, denoised, eps, state, key, process, denoise)

dataclass source

class RowKeys()

One typed PRNG key per row of a batch, keys [B], for a solver whose draws are row by row (RowSolver): a row’s draws are its own key’s, whatever rows sample beside it, as they are sampled alone, while the model still runs once over the batch.

class source

class RowSolver(Solver[StateT], Protocol[StateT])

A solver that draws each row by its own key (RowKeys): sample hands step_rows the step’s row keys where it hands step one key.

def step_rows(
x,
t,
t_next,
denoised,
eps,
state,
keys: jax.Array,
process,
denoise,
/,
) -> tuple[jax.Array, StateT]

step, row i drawing from keys[i] alone.

class source

class Solver(Protocol[StateT])

The interface for one diffusion solver step and the state it keeps between steps.

StateT is that state: nothing for a one-step solver, the previous model outputs for a multistep one. It is a type parameter, so each solver’s own state type is checked at its call sites.

A solver is a pytree, as a pipeline call takes it as data: a flax.struct.dataclass whose numbers are fields and whose settings are struct.field(pytree_node=False), and a solver wrapping another holds it as a field.

def init(x, times, process, *, key) -> StateT

Return the initial state for sampling from x over the concrete grid times.

It checks the grid’s endpoint domains at compile time. key is the root key of the whole sampling run, or a RowSolver’s RowKeys; only DPMSolverSDE reads it, for its whole-trajectory Brownian tree. Each step draws from the folded key passed to it.

def step(
x,
t,
t_next,
denoised,
eps,
state,
key,
process,
denoise,
/,
) -> tuple[jax.Array, StateT]

Return the sample at t_next and the new state, from x and the model’s (denoised, eps) at t.

sample passes every argument by position, so a solver over another algebra can name the pair for what it reads: the discrete solver takes log-probabilities where a Gaussian one takes eps.

dataclass source

class UniPC(
order: int = struct.field(pytree_node=False, default=2),
solver_type: Literal['bh1', 'bh2'] = struct.field(pytree_node=False, default='bh2'),
predict_x0: bool = struct.field(pytree_node=False, default=True),
lower_order_final: bool = struct.field(pytree_node=False, default=True),
disable_corrector: tuple[int, ...] = struct.field(pytree_node=False, default=()),
)

UniPC, a unified predictor and corrector in lambda, as Diffusers 0.34.0’s UniPCMultistepScheduler.

UniPC is from Zhao et al. (2023, arXiv 2302.04867). Its weights solve a small linear system over the history’s positions. Each step first corrects the sample the last predictor produced, with the model output just read there and that predictor’s order, then predicts the next sample from the corrected one. solver_type picks B(h) as h (bh1) or e^h - 1 (bh2); predict_x0 integrates the clean prediction, otherwise eps. lower_order_final caps the order by the steps remaining, Diffusers’ rule, so the last step is first order; disable_corrector names the step indices whose predictor’s output is not corrected. The lowered clean-prediction terminal is alpha_t*x_0; epsilon prediction uses the corrected sample with the pre-correction epsilon. Initialization rejects an unlowered higher-order zero target. At an alpha=0 source, bh1 and epsilon correctors diverge unless the first correction is disabled. The finite infinite-node weights use the same Vandermonde system with column scaling.

Every coefficient depends on the grid alone, so init solves them in float64 for each interval and order, and a step reads its interval’s row by where t sits on that grid and its order from the state. A step’s t is a point of the grid init was given.

def init(x, times, process, *, key)
def step(x, t, t_next, denoised, eps, state, key, process, denoise)