Skip to content

dew.averaging

Weight averages: the EMA’s update, and post-hoc EMA’s averages of any length, reconstructed after training.

The math the optimizer’s averages (dew.optim.power_profiles), the training step’s EMA (dew.training.transaction) and a checkpoint’s reconstruction (Checkpoints.posthoc_ema) share, below all three, so that none imports another for it.

Post-hoc EMA

This follows Karras et al., “Analyzing and Improving the Training Dynamics of Diffusion Models” (CVPR 2024, arXiv:2312.02696), Section 3 and Appendix C. After a run has made T updates, a power-function EMA weighs the weights of update t in proportion to t^γ. Its length is given by std, the standard deviation of that profile relative to T (σ_rel in the paper). The profile stretches with T, which an exponential EMA’s fixed decay does not, so one σ_rel means the same thing at every point of a run.

A run tracks a few such averages (dew.optim.power_profiles, OptimConfig.ema_profiles), and every checkpoint save also keeps a snapshot of them (Checkpoints.profile_steps). Each snapshot is a linear combination of the weights along the trajectory, with a known profile. So the average for any σ_rel at any snapshot’s step is approximately a weighted sum of snapshots, with weights from a least-squares fit of the profiles. coefficients solves for those weights (Algorithm 3), and Checkpoints.posthoc_ema sums the snapshots.

NameSummary
coefficientsReturn the weights that combine the averages in snapshots into the average of std after updates updates (Algorithm 3).
ema_updateUpdate the EMA leaves toward params, keeping each leaf’s storage dtype.
exponentReturn the exponent γ of the power-function profile whose relative standard deviation is std.
power_decayReturn the decay schedule of the power-function EMA whose relative standard deviation is std, in the form an EMASpec reads.
powered_decayThe decay schedule of the power-function EMA whose γ + 1 is power, a number or an array (power_decay).

function source

def coefficients(
snapshots: Sequence[tuple[int, float]],
updates: int,
std: float,
) -> np.ndarray

Return the weights that combine the averages in snapshots into the average of std after updates updates (Algorithm 3).

Each snapshot is a pair (updates it had seen, its relative standard deviation). The weights are the least-squares fit of the target profile by the snapshots’ profiles, normalized to sum to one. An empty snapshots, or a snapshot taken after no updates or after more than updates, raises ValueError.

function source

def ema_update(ema, params, decay)

Update the EMA leaves toward params, keeping each leaf’s storage dtype.

Each leaf of ema becomes decay * average + (1 - decay) * live, and a leaf the step left as it was keeps its average bit for bit wherever that sum is finite, however a program fuses it (a select, not a difference that finite leaves of opposite signs overflow; a NaN decay stays NaN); the live value is the leaf of params at the same path; leaves that ema does not hold are not averaged. The arithmetic runs in fp32 or wider, so fp64 leaves keep fp64 precision. A decay of 1 or more returns the average leaf as it was, even when it holds nonfinite values, as a frozen reference can.

function source

def exponent(std: float) -> float

Return the exponent γ of the power-function profile whose relative standard deviation is std.

γ is the largest real root of γ³ + 7γ² + (16 - std⁻²)γ + (12 - std⁻²) (Eq. 126 and Algorithm 2). std must lie in (0, 12^-0.5); any other value raises ValueError.

function source

def power_decay(std: float) -> optax.Schedule

Return the decay schedule of the power-function EMA whose relative standard deviation is std, in the form an EMASpec reads.

After count completed updates, update count + 1 keeps (1 - 1/(count + 1))^(γ + 1) of the average (Eq. 127). So the first update replaces the average with the weights entirely.

The decay is computed as exp((γ + 1) log1p(-1/t)). That keeps 1 - decay, which is of order (γ + 1)/t, accurate in fp32 late in a run.

function source

def powered_decay(power) -> optax.Schedule

The decay schedule of the power-function EMA whose γ + 1 is power, a number or an array (power_decay).