Skip to content

dew.diffusion.schedules

The noise schedules, one module per family.

NameSummary
ContinuousNoiseSchedulerA schedule whose time is a fraction of the trajectory, not an index.
CosineContinuousNoiseSchedulerFlaxDiff’s continuous cosine schedule, variance preserving: alpha = cos(pi t / 2) and sigma = sin(pi t / 2), each step weighted 1 / (1 + SNR), which is sigma^2.
CosineGeneralNoiseSchedulerFlaxDiff’s tangent grid of sigmas, variance exploding: sigma(t) = tan(theta(t)) / kappa, theta moving linearly from atan(sigma_min / kappa) at t = 0 to atan(sigma_max / kappa) at t = 1, the angles of FlaxDiff’s log-SNR bounds.
CosineNoiseSchedulerThe cosine beta table of Nichol and Dhariwal 2021.
DiscreteNoiseSchedulerA variance preserving schedule tabulated from betas, DDPM style.
EDMNoiseSchedulerDraws training sigmas from exp(N(P_mean, P_std^2)).
ExpNoiseSchedulerFlaxDiff’s exponential beta table (exp_beta_schedule); the P2 weight arguments are those of DiscreteNoiseScheduler.
FlowMatchingSchedulerRectified flow / conditional flow matching on the linear path.
GeneralizedNoiseSchedulerThe variance-exploding schedule family of the EDM paper.
KarrasVENoiseSchedulerPlaces sigmas along t with the rho spacing of the EDM paper.
LinearNoiseSchedulerThe linear beta table of DDPM, scaled to the step count.
NoiseSchedulerDefines a forward process x_t = alpha(t) x_0 + sigma(t) eps on the time domain [0, T].
RectifiedFlowA schedule on the rectified-flow path, x_t = (1 - sigma) x_0 + sigma epsilon: its rates(t) are (1 - sigma, sigma).
SqrtContinuousNoiseSchedulerSquare-root schedule from Diffusion-LM.
cosine_beta_scheduleReturn the cosine beta table of Nichol and Dhariwal 2021.
exp_beta_scheduleReturn FlaxDiff’s exponential betas: the cumulative alpha is exp(-12 t) on t = i / T, T = timesteps, and beta_i = 1 - alpha_bar((i + 1) / T) / alpha_bar(i / T), clipped to [0, beta_end].
expandReturn a per-example coefficient [B] reshaped to broadcast against x [B, ...].
linear_beta_scheduleReturn DDPM’s betas, linear from beta_start to beta_end over 1000 steps.

class source

class ContinuousNoiseScheduler(NoiseScheduler)

A schedule whose time is a fraction of the trajectory, not an index.

T is 1.0, so t = 1 is fully noised, and training draws t uniformly. A subclass provides the rates and the weight of its parameterization.

def sample_t(key, n)

class source

class CosineContinuousNoiseScheduler(ContinuousNoiseScheduler)

FlaxDiff’s continuous cosine schedule, variance preserving: alpha = cos(pi t / 2) and sigma = sin(pi t / 2), each step weighted 1 / (1 + SNR), which is sigma^2. The arithmetic is FlaxDiff’s CosineContinuousNoiseScheduler (flaxdiff/schedulers/cosine.py at 15c55b001), operation for operation, in float32 or a float64 run’s float64 (times).

def rates(t)
def weight(t)

class source

class CosineGeneralNoiseScheduler(
sigma_min: float = 0.02,
sigma_max: float = 80.0,
kappa: float = 1.0,
sigma_data: float = 1.0,
)

FlaxDiff’s tangent grid of sigmas, variance exploding: sigma(t) = tan(theta(t)) / kappa, theta moving linearly from atan(sigma_min / kappa) at t = 0 to atan(sigma_max / kappa) at t = 1, the angles of FlaxDiff’s log-SNR bounds. With kappa 1 the grid runs from sigma_min to sigma_max; with another it runs from sigma_min / kappa^2 to sigma_max / kappa^2, as FlaxDiff’s divides the tangent by kappa again: those ends are its sigma_range, which a solver keeps its sigmas inside (Heun’s churn, BrownianTree), and sigma_min and sigma_max the arguments, as FlaxDiff keeps them and a record rebuilds the schedule from.

The arithmetic is FlaxDiff’s CosineGeneralNoiseScheduler and the GeneralizedNoiseScheduler it extends (flaxdiff/schedulers/cosine.py and common.py at 15c55b001): its grid operation for operation, each step weighted 1 + sigma^2 / sigma_max^2, its nominal sigma_max (weight), the model conditioned on the step’s progress in whole thousandths (model_time), and sigma_data 1, not the EDM family’s weight, log-sigma conditioning and 0.5. In float32, or a float64 run’s float64 (times). The weight is FlaxDiff’s function written without its cancellation: FlaxDiff forms sigma^2 as 1 / (1 + (1 - sigma^2) / sigma^2), whose sum cancels as sigma grows, so its float32 weight is 4e-4 off at the default grid’s top and 2% off at a sigma of 640.

t_of_sigma, which FlaxDiff leaves unwritten, takes the angle above the grid’s bottom as one arctangent, atan(a) - atan(b) = atan((a - b) / (1 + a b)) for a b > 0, where the formula’s difference of two angles cancels near the bottom (1.6 times as far from float64 as the formula with each operation rounded to float32 at a kappa of 0.5, 1.25 times as one arctangent), and divides as IEEE division does (divided), not as XLA’s product with the reciprocal of the constant.

def sigmas(t)
def t_of_sigma(sigma)
def weight(t)
def model_time(t)

class source

class CosineNoiseScheduler(
timesteps: int,
beta_start: float = 0.008,
beta_end: float = 0.999,
p2_loss_weight_k: float = 1,
p2_loss_weight_gamma: float = 1,
)

The cosine beta table of Nichol and Dhariwal 2021.

beta_start is the offset s of cosine_beta_schedule, and beta_end is the value each beta is clipped at. The P2 weight arguments are those of DiscreteNoiseScheduler.

class source

class DiscreteNoiseScheduler(
betas: np.ndarray | Sequence[float],
p2_loss_weight_k: float = 1,
p2_loss_weight_gamma: float = 1,
)

A variance preserving schedule tabulated from betas, DDPM style.

signal_rate^2 + noise_rate^2 = 1 at every index, and t is the index into the table, so T is the number of entries. The loss weight is the P2 weight of Choi et al. 2022, (k + SNR)^-gamma, with k = p2_loss_weight_k and gamma = p2_loss_weight_gamma. At the defaults k = 1, gamma = 1 it is 1 / (1 + SNR). On a v-prediction loss, whose error is 1 + SNR times the x_0 error, that is exactly an unweighted x_0 loss. All fixed tables, including the weight, are prepared in host float64 and rounded once.

def index(t) -> jax.Array

Return t as a table index.

A time grid may reach T itself, which maps to the last entry.

def rates(t)
def variances(t)
def sample_t(key, n)
def weight(t)

class source

class EDMNoiseScheduler(
sigma_min: float = 0.002,
sigma_max: float = 80.0,
sigma_data: float = 0.5,
P_mean: float = -0.4,
P_std: float = 1.0,
)

Draws training sigmas from exp(N(P_mean, P_std^2)).

t is the standard normal draw, and sigma(t) = exp(P_mean + P_std t). The defaults are EDM2’s (Karras et al. 2024), because EDM1’s -1.2/1.2 concentrated too much mass on low noise levels for larger models. Pass P_mean=-1.2 and P_std=1.2 explicitly to reproduce an EDM1 run.

def sigmas(t)
def t_of_sigma(sigma)
def sample_t(key, n)

class source

class ExpNoiseScheduler(
timesteps: int,
beta_start: float = 0.008,
beta_end: float = 0.999,
p2_loss_weight_k: float = 1,
p2_loss_weight_gamma: float = 1,
)

FlaxDiff’s exponential beta table (exp_beta_schedule); the P2 weight arguments are those of DiscreteNoiseScheduler.

class source

class FlowMatchingScheduler(
shift: float = 1.0,
logit_mean: float = 0.0,
logit_std: float = 1.0,
density: Density = 'logit_normal',
mode_scale: float = 1.29,
dynamic: bool = False,
)

Rectified flow / conditional flow matching on the linear path.

x_t = (1 - t) * x_0 + t * epsilon for t in [0, 1], so alpha + sigma = 1 and the model input needs no scaling. Training times are drawn from one of SD3’s densities (density):

  • logit_normal: t = sigmoid(logit_mean + logit_std * n), n ~ N(0, 1), which concentrates training on the middle of the trajectory, where the velocity is hardest to predict.
  • mode: t = 1 - u - s (cos^2(pi u / 2) - 1 + u), u ~ U(0, 1), with s = mode_scale in [-1, 2 / (pi - 2)] (Eq. 20). This density has a mode in the middle, keeps weight on both ends, and is uniform at s = 0.
  • cosmap: t = 1 - 1 / (tan(pi u / 2) + 1) (Eq. 21).
  • uniform: t = u.

Training draws the unshifted time. shift then maps it, and the sampling grid, to the resolution’s noise levels with shift t / (1 + (shift - 1) t), and the model is conditioned on the shifted time times 1000. A dynamic shift, one a resolution sets (ResolutionShift, a scheduler file’s dynamic shifting), takes that map as Diffusers’ dynamic shifting does, shift / (shift + 1 / t - 1); a static one as its static schedule does. The two are one function and round apart, each as its source rounds it. An unknown density, or a mode_scale outside its range for mode, raises ValueError.

def shift_timesteps(t) -> jax.Array

t at the shift, operation for operation as Diffusers maps it: a dynamic shift as FlowMatchEulerDiscreteScheduler.time_shift, a static one as its static sigmas.

def sample_t(key, n)
def rates(t)
def weight(t)
def model_time(t)

class source

class GeneralizedNoiseScheduler(
sigma_min: float = 0.002,
sigma_max: float = 80.0,
sigma_data: float = 0.5,
)

The variance-exploding schedule family of the EDM paper.

The paper is Karras et al. 2022, “Elucidating the Design Space of Diffusion-Based Generative Models”. alpha is 1, and the paired preconditioning scales the model input. Every member conditions the model on c_noise = log(sigma) / 4 and weights the loss with lambda(sigma) = (sigma^2 + sigma_data^2) / (sigma sigma_data)^2 (Eq. 8 of the paper). A subclass places the sigmas along t (sigmas) and inverts that placement (t_of_sigma) for the solvers that step in sigma.

sigma_range: tuple[float, float]

The least and greatest sigma the schedule’s grid reaches, which a solver keeps its sigmas inside (Heun’s churn, the Brownian tree): sigma_min and sigma_max where the grid runs between its arguments, as every member’s but FlaxDiff’s tangent grid does.

def sigmas(t) -> jax.Array

Return the noise level at t.

def t_of_sigma(sigma) -> jax.Array

Return the time at which sigmas gives sigma, the inverse of sigmas.

def rates(t)
def weight(t)
def model_time(t)

class source

class KarrasVENoiseScheduler(
sigma_min: float = 0.002,
sigma_max: float = 80.0,
rho: float = 7.0,
sigma_data: float = 0.5,
)

Places sigmas along t with the rho spacing of the EDM paper.

The spacing is Eq. 5 of Karras et al. 2022, sigma(t) = (sigma_max^(1/rho) + (1 - t) (sigma_min^(1/rho) - sigma_max^(1/rho)))^rho, so a uniform grid in t is the paper’s sampling grid in sigma.

def sigmas(t)
def t_of_sigma(sigma)

class source

class LinearNoiseScheduler(
timesteps: int,
beta_start: float = 0.0001,
beta_end: float = 0.02,
p2_loss_weight_k: float = 1,
p2_loss_weight_gamma: float = 1,
)

The linear beta table of DDPM, scaled to the step count.

DDPM is Ho et al. 2020.

class source

class NoiseScheduler(ABC)

Defines a forward process x_t = alpha(t) x_0 + sigma(t) eps on the time domain [0, T].

t = T is the fully noised end.

def rates(t) -> tuple[jax.Array, jax.Array]

Return (alpha, sigma) at t, shaped like t.

def sample_t(key, n: int) -> jax.Array

Draw n training times from this schedule’s training distribution, independent and alike: an objective draws each example’s time as sample_t(its key, 1) (Step.sample), so a draw that couples the n (stratified, antithetic) is the objective’s, over the update’s places (Step.places), as MDLM’s is.

def weight(t) -> jax.Array

Return the schedule’s own loss weight at t.

The weight is in the space where the schedule’s paired parameterization computes the loss.

def model_time(t) -> jax.Array

Return the time value the model is conditioned on at t.

By default this is t itself, and a schedule can override it.

def prior_scale() -> jax.Array

Return the standard deviation of the initial Gaussian draw.

By default this is sqrt(alpha^2 + sigma^2) at T.

def step_interval(t, t_next) -> jax.Array

Return the transfer interval from t to t_next.

An ordinary grid advances to its next point, so the interval is t - t_next.

def half_interval(t, t_next) -> jax.Array

Return half a grid interval, where a Runge-Kutta stage places its intermediate points.

def variances(t) -> tuple[jax.Array, jax.Array]

Return (alpha^2, sigma^2) at t, shaped like t: the rates squared, unless the schedule holds the squares themselves, as a beta table’s cumulative products are, whose differences a transfer between two of them reads more exactly than the squares of its rounded roots (DiscreteNoiseScheduler).

def snr(t) -> jax.Array

class source

class RectifiedFlow

A schedule on the rectified-flow path, x_t = (1 - sigma) x_0 + sigma epsilon: its rates(t) are (1 - sigma, sigma). A sampler whose step holds only on that path (dew.sampling.FlowSDE) asks for this, whether the schedule is Dew’s own (FlowMatchingScheduler) or a published source’s grid (dew.diffusion.schedules.source_grids.FlowGrid).

class source

class SqrtContinuousNoiseScheduler(ContinuousNoiseScheduler)

Square-root schedule from Diffusion-LM.

Diffusion-LM is Li et al. 2022. The cumulative alpha is 1 - sqrt(t + s), s = 1e-4, normalized to one at t = 0 the way Diffusion-LM’s betas_for_alpha_bar normalizes it:

alpha^2(t) = (1 - sqrt(t + s)) / (1 - sqrt(s)), sigma^2 = 1 - alpha^2,

so at Diffusion-LM’s step k of T the rates are its table’s at t = (k + 1) / T. Noise rises like t^(1/4) from t = 0, much faster than in the cosine schedule, so fewer steps are spent where an embedding carries little noise. The cumulative alpha reaches zero just before t = 1, and the rates stay at (0, 1) from there; Diffusion-LM’s last step instead clips its beta at 0.999. At t >= 1 - s, alpha’s derivative in t is infinite, so an objective that differentiates the rates in time (MeanFlow, consistency training) must not draw t from the last 1e-4. The paper trains the plain x_0 loss, so the weight is one.

def rates(t)
def weight(t)

function source

def cosine_beta_schedule(timesteps, start_angle=0.008, end_angle=0.999)

Return the cosine beta table of Nichol and Dhariwal 2021.

The table is their Eq. 17. The cumulative alpha follows f(t) = cos^2((t + s) / (1 + s) * pi / 2) at t = i / T, with T = timesteps and s = start_angle, and beta_i = 1 - f((i + 1) / T) / f(i / T), clipped at end_angle. The arithmetic is improved-diffusion’s betas_for_alpha_bar, operation for operation and in scalar math.cos (NumPy’s vectorized cosine rounds a few arguments differently), so the table matches the authors’ to the bit.

function source

def exp_beta_schedule(timesteps, beta_start=0.008, beta_end=0.999)

Return FlaxDiff’s exponential betas: the cumulative alpha is exp(-12 t) on t = i / T, T = timesteps, and beta_i = 1 - alpha_bar((i + 1) / T) / alpha_bar(i / T), clipped to [0, beta_end].

beta_start is FlaxDiff’s argument, which its table reads nothing of. The arithmetic is FlaxDiff’s exp_beta_schedule (flaxdiff/schedulers/exp.py at 15c55b001), operation for operation in float64 NumPy, so the table matches its to the bit; Diffusers’ exponential table, which caps every beta at 0.999 whatever its beta_end, is source_grids.published_betas’.

function source

def expand(coefficient, x)

Return a per-example coefficient [B] reshaped to broadcast against x [B, ...].

function source

def linear_beta_schedule(timesteps, beta_start=0.0001, beta_end=0.02)

Return DDPM’s betas, linear from beta_start to beta_end over 1000 steps.

The table is Ho et al. 2020’s. For another step count, the betas are scaled by 1000 / timesteps so the cumulative alpha stays roughly the same.