dew.diffusion.schedules
The noise schedules, one module per family.
| Name | Summary |
|---|---|
ContinuousNoiseScheduler | A schedule whose time is a fraction of the trajectory, not an index. |
CosineContinuousNoiseScheduler | FlaxDiff’s continuous cosine schedule, variance preserving: alpha = cos(pi t / 2) and sigma = sin(pi t / 2), each step weighted 1 / (1 + SNR), which is sigma^2. |
CosineGeneralNoiseScheduler | FlaxDiff’s tangent grid of sigmas, variance exploding: sigma(t) = tan(theta(t)) / kappa, theta moving linearly from atan(sigma_min / kappa) at t = 0 to atan(sigma_max / kappa) at t = 1, the angles of FlaxDiff’s log-SNR bounds. |
CosineNoiseScheduler | The cosine beta table of Nichol and Dhariwal 2021. |
DiscreteNoiseScheduler | A variance preserving schedule tabulated from betas, DDPM style. |
EDMNoiseScheduler | Draws training sigmas from exp(N(P_mean, P_std^2)). |
ExpNoiseScheduler | FlaxDiff’s exponential beta table (exp_beta_schedule); the P2 weight arguments are those of DiscreteNoiseScheduler. |
FlowMatchingScheduler | Rectified flow / conditional flow matching on the linear path. |
GeneralizedNoiseScheduler | The variance-exploding schedule family of the EDM paper. |
KarrasVENoiseScheduler | Places sigmas along t with the rho spacing of the EDM paper. |
LinearNoiseScheduler | The linear beta table of DDPM, scaled to the step count. |
NoiseScheduler | Defines a forward process x_t = alpha(t) x_0 + sigma(t) eps on the time domain [0, T]. |
RectifiedFlow | A schedule on the rectified-flow path, x_t = (1 - sigma) x_0 + sigma epsilon: its rates(t) are (1 - sigma, sigma). |
SqrtContinuousNoiseScheduler | Square-root schedule from Diffusion-LM. |
cosine_beta_schedule | Return the cosine beta table of Nichol and Dhariwal 2021. |
exp_beta_schedule | Return FlaxDiff’s exponential betas: the cumulative alpha is exp(-12 t) on t = i / T, T = timesteps, and beta_i = 1 - alpha_bar((i + 1) / T) / alpha_bar(i / T), clipped to [0, beta_end]. |
expand | Return a per-example coefficient [B] reshaped to broadcast against x [B, ...]. |
linear_beta_schedule | Return DDPM’s betas, linear from beta_start to beta_end over 1000 steps. |
ContinuousNoiseScheduler
Section titled “ContinuousNoiseScheduler”class ContinuousNoiseScheduler(NoiseScheduler)A schedule whose time is a fraction of the trajectory, not an index.
T is 1.0, so t = 1 is fully noised, and training draws t uniformly. A subclass provides the rates and the weight of its parameterization.
ContinuousNoiseScheduler.sample_t
Section titled “ContinuousNoiseScheduler.sample_t”def sample_t(key, n)CosineContinuousNoiseScheduler
Section titled “CosineContinuousNoiseScheduler”class CosineContinuousNoiseScheduler(ContinuousNoiseScheduler)FlaxDiff’s continuous cosine schedule, variance preserving: alpha = cos(pi t / 2) and
sigma = sin(pi t / 2), each step weighted 1 / (1 + SNR), which is sigma^2. The arithmetic is
FlaxDiff’s CosineContinuousNoiseScheduler (flaxdiff/schedulers/cosine.py at 15c55b001),
operation for operation, in float32 or a float64 run’s float64 (times).
CosineContinuousNoiseScheduler.rates
Section titled “CosineContinuousNoiseScheduler.rates”def rates(t)CosineContinuousNoiseScheduler.weight
Section titled “CosineContinuousNoiseScheduler.weight”def weight(t)CosineGeneralNoiseScheduler
Section titled “CosineGeneralNoiseScheduler”class CosineGeneralNoiseScheduler( sigma_min: float = 0.02, sigma_max: float = 80.0, kappa: float = 1.0, sigma_data: float = 1.0,)FlaxDiff’s tangent grid of sigmas, variance exploding: sigma(t) = tan(theta(t)) / kappa, theta
moving linearly from atan(sigma_min / kappa) at t = 0 to atan(sigma_max / kappa) at t = 1, the
angles of FlaxDiff’s log-SNR bounds. With kappa 1 the grid runs from sigma_min to sigma_max; with
another it runs from sigma_min / kappa^2 to sigma_max / kappa^2, as FlaxDiff’s divides the tangent
by kappa again: those ends are its sigma_range, which a solver keeps its sigmas inside (Heun’s
churn, BrownianTree), and sigma_min and sigma_max the arguments, as FlaxDiff keeps them and a
record rebuilds the schedule from.
The arithmetic is FlaxDiff’s CosineGeneralNoiseScheduler and the GeneralizedNoiseScheduler
it extends (flaxdiff/schedulers/cosine.py and common.py at 15c55b001): its grid operation for
operation, each step weighted 1 + sigma^2 / sigma_max^2, its nominal sigma_max (weight), the
model conditioned on the step’s progress in whole thousandths (model_time), and sigma_data 1,
not the EDM family’s weight, log-sigma conditioning and 0.5. In float32, or a float64 run’s
float64 (times). The weight is FlaxDiff’s function written without its cancellation: FlaxDiff
forms sigma^2 as 1 / (1 + (1 - sigma^2) / sigma^2), whose sum cancels as sigma grows, so its
float32 weight is 4e-4 off at the default grid’s top and 2% off at a sigma of 640.
t_of_sigma, which FlaxDiff leaves unwritten, takes the angle above the grid’s bottom as one
arctangent, atan(a) - atan(b) = atan((a - b) / (1 + a b)) for a b > 0, where the formula’s
difference of two angles cancels near the bottom (1.6 times as far from float64 as the formula
with each operation rounded to float32 at a kappa of 0.5, 1.25 times as one arctangent), and
divides as IEEE division does (divided), not as XLA’s product with the reciprocal of the
constant.
CosineGeneralNoiseScheduler.sigmas
Section titled “CosineGeneralNoiseScheduler.sigmas”def sigmas(t)CosineGeneralNoiseScheduler.t_of_sigma
Section titled “CosineGeneralNoiseScheduler.t_of_sigma”def t_of_sigma(sigma)CosineGeneralNoiseScheduler.weight
Section titled “CosineGeneralNoiseScheduler.weight”def weight(t)CosineGeneralNoiseScheduler.model_time
Section titled “CosineGeneralNoiseScheduler.model_time”def model_time(t)CosineNoiseScheduler
Section titled “CosineNoiseScheduler”class CosineNoiseScheduler( timesteps: int, beta_start: float = 0.008, beta_end: float = 0.999, p2_loss_weight_k: float = 1, p2_loss_weight_gamma: float = 1,)The cosine beta table of Nichol and Dhariwal 2021.
beta_start is the offset s of cosine_beta_schedule, and beta_end
is the value each beta is clipped at. The P2 weight arguments are those
of DiscreteNoiseScheduler.
DiscreteNoiseScheduler
Section titled “DiscreteNoiseScheduler”class DiscreteNoiseScheduler( betas: np.ndarray | Sequence[float], p2_loss_weight_k: float = 1, p2_loss_weight_gamma: float = 1,)A variance preserving schedule tabulated from betas, DDPM style.
signal_rate^2 + noise_rate^2 = 1 at every index, and t is the index into
the table, so T is the number of entries. The loss weight is the P2
weight of Choi et al. 2022, (k + SNR)^-gamma, with
k = p2_loss_weight_k and gamma = p2_loss_weight_gamma. At the
defaults k = 1, gamma = 1 it is 1 / (1 + SNR). On a v-prediction loss,
whose error is 1 + SNR times the x_0 error, that is exactly an
unweighted x_0 loss. All fixed tables, including the weight, are
prepared in host float64 and rounded once.
DiscreteNoiseScheduler.index
Section titled “DiscreteNoiseScheduler.index”def index(t) -> jax.ArrayReturn t as a table index.
A time grid may reach T itself, which maps to the last entry.
DiscreteNoiseScheduler.rates
Section titled “DiscreteNoiseScheduler.rates”def rates(t)DiscreteNoiseScheduler.variances
Section titled “DiscreteNoiseScheduler.variances”def variances(t)DiscreteNoiseScheduler.sample_t
Section titled “DiscreteNoiseScheduler.sample_t”def sample_t(key, n)DiscreteNoiseScheduler.weight
Section titled “DiscreteNoiseScheduler.weight”def weight(t)EDMNoiseScheduler
Section titled “EDMNoiseScheduler”class EDMNoiseScheduler( sigma_min: float = 0.002, sigma_max: float = 80.0, sigma_data: float = 0.5, P_mean: float = -0.4, P_std: float = 1.0,)Draws training sigmas from exp(N(P_mean, P_std^2)).
t is the standard normal draw, and sigma(t) = exp(P_mean + P_std t). The defaults are EDM2’s (Karras et al. 2024), because EDM1’s -1.2/1.2 concentrated too much mass on low noise levels for larger models. Pass P_mean=-1.2 and P_std=1.2 explicitly to reproduce an EDM1 run.
EDMNoiseScheduler.sigmas
Section titled “EDMNoiseScheduler.sigmas”def sigmas(t)EDMNoiseScheduler.t_of_sigma
Section titled “EDMNoiseScheduler.t_of_sigma”def t_of_sigma(sigma)EDMNoiseScheduler.sample_t
Section titled “EDMNoiseScheduler.sample_t”def sample_t(key, n)ExpNoiseScheduler
Section titled “ExpNoiseScheduler”class ExpNoiseScheduler( timesteps: int, beta_start: float = 0.008, beta_end: float = 0.999, p2_loss_weight_k: float = 1, p2_loss_weight_gamma: float = 1,)FlaxDiff’s exponential beta table (exp_beta_schedule); the P2 weight arguments are those of
DiscreteNoiseScheduler.
FlowMatchingScheduler
Section titled “FlowMatchingScheduler”class FlowMatchingScheduler( shift: float = 1.0, logit_mean: float = 0.0, logit_std: float = 1.0, density: Density = 'logit_normal', mode_scale: float = 1.29, dynamic: bool = False,)Rectified flow / conditional flow matching on the linear path.
x_t = (1 - t) * x_0 + t * epsilon for t in [0, 1], so alpha + sigma = 1 and
the model input needs no scaling. Training times are drawn from one of
SD3’s densities (density):
logit_normal: t = sigmoid(logit_mean + logit_std * n), n ~ N(0, 1), which concentrates training on the middle of the trajectory, where the velocity is hardest to predict.mode: t = 1 - u - s (cos^2(pi u / 2) - 1 + u), u ~ U(0, 1), with s =mode_scalein [-1, 2 / (pi - 2)] (Eq. 20). This density has a mode in the middle, keeps weight on both ends, and is uniform at s = 0.cosmap: t = 1 - 1 / (tan(pi u / 2) + 1) (Eq. 21).uniform: t = u.
Training draws the unshifted time. shift then maps it, and the
sampling grid, to the resolution’s noise levels with
shift t / (1 + (shift - 1) t), and the model is conditioned on the
shifted time times 1000. A dynamic shift, one a resolution sets
(ResolutionShift, a scheduler file’s dynamic shifting), takes that map
as Diffusers’ dynamic shifting does, shift / (shift + 1 / t - 1); a
static one as its static schedule does. The two are one function and
round apart, each as its source rounds it. An unknown density, or a
mode_scale outside its range for mode, raises ValueError.
FlowMatchingScheduler.shift_timesteps
Section titled “FlowMatchingScheduler.shift_timesteps”def shift_timesteps(t) -> jax.Arrayt at the shift, operation for operation as Diffusers maps it: a dynamic shift as
FlowMatchEulerDiscreteScheduler.time_shift, a static one as its static sigmas.
FlowMatchingScheduler.sample_t
Section titled “FlowMatchingScheduler.sample_t”def sample_t(key, n)FlowMatchingScheduler.rates
Section titled “FlowMatchingScheduler.rates”def rates(t)FlowMatchingScheduler.weight
Section titled “FlowMatchingScheduler.weight”def weight(t)FlowMatchingScheduler.model_time
Section titled “FlowMatchingScheduler.model_time”def model_time(t)GeneralizedNoiseScheduler
Section titled “GeneralizedNoiseScheduler”class GeneralizedNoiseScheduler( sigma_min: float = 0.002, sigma_max: float = 80.0, sigma_data: float = 0.5,)The variance-exploding schedule family of the EDM paper.
The paper is Karras et al. 2022, “Elucidating the Design Space of
Diffusion-Based Generative Models”. alpha is 1, and the paired
preconditioning scales the model input. Every member conditions the
model on c_noise = log(sigma) / 4 and weights the loss with
lambda(sigma) = (sigma^2 + sigma_data^2) / (sigma sigma_data)^2 (Eq. 8
of the paper). A subclass places the sigmas along t (sigmas) and
inverts that placement (t_of_sigma) for the solvers that step in
sigma.
sigma_range: tuple[float, float]-
The least and greatest sigma the schedule’s grid reaches, which a solver keeps its sigmas inside (Heun’s churn, the Brownian tree):
sigma_minandsigma_maxwhere the grid runs between its arguments, as every member’s but FlaxDiff’s tangent grid does.
GeneralizedNoiseScheduler.sigmas
Section titled “GeneralizedNoiseScheduler.sigmas”def sigmas(t) -> jax.ArrayReturn the noise level at t.
GeneralizedNoiseScheduler.t_of_sigma
Section titled “GeneralizedNoiseScheduler.t_of_sigma”def t_of_sigma(sigma) -> jax.ArrayReturn the time at which sigmas gives sigma, the inverse of sigmas.
GeneralizedNoiseScheduler.rates
Section titled “GeneralizedNoiseScheduler.rates”def rates(t)GeneralizedNoiseScheduler.weight
Section titled “GeneralizedNoiseScheduler.weight”def weight(t)GeneralizedNoiseScheduler.model_time
Section titled “GeneralizedNoiseScheduler.model_time”def model_time(t)KarrasVENoiseScheduler
Section titled “KarrasVENoiseScheduler”class KarrasVENoiseScheduler( sigma_min: float = 0.002, sigma_max: float = 80.0, rho: float = 7.0, sigma_data: float = 0.5,)Places sigmas along t with the rho spacing of the EDM paper.
The spacing is Eq. 5 of Karras et al. 2022, sigma(t) = (sigma_max^(1/rho) + (1 - t) (sigma_min^(1/rho) - sigma_max^(1/rho)))^rho, so a uniform grid in t is the paper’s sampling grid in sigma.
KarrasVENoiseScheduler.sigmas
Section titled “KarrasVENoiseScheduler.sigmas”def sigmas(t)KarrasVENoiseScheduler.t_of_sigma
Section titled “KarrasVENoiseScheduler.t_of_sigma”def t_of_sigma(sigma)LinearNoiseScheduler
Section titled “LinearNoiseScheduler”class LinearNoiseScheduler( timesteps: int, beta_start: float = 0.0001, beta_end: float = 0.02, p2_loss_weight_k: float = 1, p2_loss_weight_gamma: float = 1,)The linear beta table of DDPM, scaled to the step count.
DDPM is Ho et al. 2020.
NoiseScheduler
Section titled “NoiseScheduler”class NoiseScheduler(ABC)Defines a forward process x_t = alpha(t) x_0 + sigma(t) eps on the time domain [0, T].
t = T is the fully noised end.
NoiseScheduler.rates
Section titled “NoiseScheduler.rates”def rates(t) -> tuple[jax.Array, jax.Array]Return (alpha, sigma) at t, shaped like t.
NoiseScheduler.sample_t
Section titled “NoiseScheduler.sample_t”def sample_t(key, n: int) -> jax.ArrayDraw n training times from this schedule’s training distribution, independent and alike:
an objective draws each example’s time as sample_t(its key, 1) (Step.sample), so a draw
that couples the n (stratified, antithetic) is the objective’s, over the update’s places
(Step.places), as MDLM’s is.
NoiseScheduler.weight
Section titled “NoiseScheduler.weight”def weight(t) -> jax.ArrayReturn the schedule’s own loss weight at t.
The weight is in the space where the schedule’s paired parameterization computes the loss.
NoiseScheduler.model_time
Section titled “NoiseScheduler.model_time”def model_time(t) -> jax.ArrayReturn the time value the model is conditioned on at t.
By default this is t itself, and a schedule can override it.
NoiseScheduler.prior_scale
Section titled “NoiseScheduler.prior_scale”def prior_scale() -> jax.ArrayReturn the standard deviation of the initial Gaussian draw.
By default this is sqrt(alpha^2 + sigma^2) at T.
NoiseScheduler.step_interval
Section titled “NoiseScheduler.step_interval”def step_interval(t, t_next) -> jax.ArrayReturn the transfer interval from t to t_next.
An ordinary grid advances to its next point, so the interval is
t - t_next.
NoiseScheduler.half_interval
Section titled “NoiseScheduler.half_interval”def half_interval(t, t_next) -> jax.ArrayReturn half a grid interval, where a Runge-Kutta stage places its intermediate points.
NoiseScheduler.variances
Section titled “NoiseScheduler.variances”def variances(t) -> tuple[jax.Array, jax.Array]Return (alpha^2, sigma^2) at t, shaped like t: the rates squared, unless the
schedule holds the squares themselves, as a beta table’s cumulative products are, whose
differences a transfer between two of them reads more exactly than the squares of its
rounded roots (DiscreteNoiseScheduler).
NoiseScheduler.snr
Section titled “NoiseScheduler.snr”def snr(t) -> jax.ArrayRectifiedFlow
Section titled “RectifiedFlow”class RectifiedFlowA schedule on the rectified-flow path, x_t = (1 - sigma) x_0 + sigma
epsilon: its rates(t) are (1 - sigma, sigma). A sampler whose step
holds only on that path (dew.sampling.FlowSDE) asks for this, whether
the schedule is Dew’s own (FlowMatchingScheduler) or a published
source’s grid (dew.diffusion.schedules.source_grids.FlowGrid).
SqrtContinuousNoiseScheduler
Section titled “SqrtContinuousNoiseScheduler”class SqrtContinuousNoiseScheduler(ContinuousNoiseScheduler)Square-root schedule from Diffusion-LM.
Diffusion-LM is Li et al. 2022. The cumulative alpha is 1 - sqrt(t + s),
s = 1e-4, normalized to one at t = 0 the way Diffusion-LM’s
betas_for_alpha_bar normalizes it:
alpha^2(t) = (1 - sqrt(t + s)) / (1 - sqrt(s)), sigma^2 = 1 - alpha^2,so at Diffusion-LM’s step k of T the rates are its table’s at t = (k + 1) / T. Noise rises like t^(1/4) from t = 0, much faster than in the cosine schedule, so fewer steps are spent where an embedding carries little noise. The cumulative alpha reaches zero just before t = 1, and the rates stay at (0, 1) from there; Diffusion-LM’s last step instead clips its beta at 0.999. At t >= 1 - s, alpha’s derivative in t is infinite, so an objective that differentiates the rates in time (MeanFlow, consistency training) must not draw t from the last 1e-4. The paper trains the plain x_0 loss, so the weight is one.
SqrtContinuousNoiseScheduler.rates
Section titled “SqrtContinuousNoiseScheduler.rates”def rates(t)SqrtContinuousNoiseScheduler.weight
Section titled “SqrtContinuousNoiseScheduler.weight”def weight(t)cosine_beta_schedule
Section titled “cosine_beta_schedule”def cosine_beta_schedule(timesteps, start_angle=0.008, end_angle=0.999)Return the cosine beta table of Nichol and Dhariwal 2021.
The table is their Eq. 17. The cumulative alpha follows
f(t) = cos^2((t + s) / (1 + s) * pi / 2) at t = i / T, with
T = timesteps and s = start_angle, and
beta_i = 1 - f((i + 1) / T) / f(i / T), clipped at end_angle. The
arithmetic is improved-diffusion’s betas_for_alpha_bar, operation for
operation and in scalar math.cos (NumPy’s vectorized cosine rounds a
few arguments differently), so the table matches the authors’ to the
bit.
exp_beta_schedule
Section titled “exp_beta_schedule”def exp_beta_schedule(timesteps, beta_start=0.008, beta_end=0.999)Return FlaxDiff’s exponential betas: the cumulative alpha is exp(-12 t) on t = i / T, T =
timesteps, and beta_i = 1 - alpha_bar((i + 1) / T) / alpha_bar(i / T), clipped to [0, beta_end].
beta_start is FlaxDiff’s argument, which its table reads nothing of. The arithmetic is FlaxDiff’s
exp_beta_schedule (flaxdiff/schedulers/exp.py at 15c55b001), operation for operation in float64
NumPy, so the table matches its to the bit; Diffusers’ exponential table, which caps every beta at
0.999 whatever its beta_end, is source_grids.published_betas’.
expand
Section titled “expand”def expand(coefficient, x)Return a per-example coefficient [B] reshaped to broadcast against x [B, ...].
linear_beta_schedule
Section titled “linear_beta_schedule”def linear_beta_schedule(timesteps, beta_start=0.0001, beta_end=0.02)Return DDPM’s betas, linear from beta_start to beta_end over 1000 steps.
The table is Ho et al. 2020’s. For another step count, the betas are
scaled by 1000 / timesteps so the cumulative alpha stays roughly the
same.