dew.averaging
Weight averages: the EMA’s update, and post-hoc EMA’s averages of any length, reconstructed after training.
The math the optimizer’s averages (dew.optim.power_profiles), the training step’s EMA
(dew.training.transaction) and a checkpoint’s reconstruction (Checkpoints.posthoc_ema) share,
below all three, so that none imports another for it.
Post-hoc EMA
This follows Karras et al., “Analyzing and Improving the Training Dynamics
of Diffusion Models” (CVPR 2024, arXiv:2312.02696), Section 3 and Appendix
C. After a run has made T updates, a power-function EMA weighs the weights
of update t in proportion to t^γ. Its length is given by std, the
standard deviation of that profile relative to T (σ_rel in the paper). The
profile stretches with T, which an exponential EMA’s fixed decay does not,
so one σ_rel means the same thing at every point of a run.
A run tracks a few such averages (dew.optim.power_profiles,
OptimConfig.ema_profiles), and every checkpoint save also keeps a
snapshot of them (Checkpoints.profile_steps). Each snapshot is a linear
combination of the weights along the trajectory, with a known profile. So
the average for any σ_rel at any snapshot’s step is approximately a
weighted sum of snapshots, with weights from a least-squares fit of the
profiles. coefficients solves for those weights (Algorithm 3), and
Checkpoints.posthoc_ema sums the snapshots.
| Name | Summary |
|---|---|
coefficients | Return the weights that combine the averages in snapshots into the average of std after updates updates (Algorithm 3). |
ema_update | Update the EMA leaves toward params, keeping each leaf’s storage dtype. |
exponent | Return the exponent γ of the power-function profile whose relative standard deviation is std. |
power_decay | Return the decay schedule of the power-function EMA whose relative standard deviation is std, in the form an EMASpec reads. |
powered_decay | The decay schedule of the power-function EMA whose γ + 1 is power, a number or an array (power_decay). |
coefficients
Section titled “coefficients”def coefficients( snapshots: Sequence[tuple[int, float]], updates: int, std: float,) -> np.ndarrayReturn the weights that combine the averages in snapshots into the
average of std after updates updates (Algorithm 3).
Each snapshot is a pair (updates it had seen, its relative standard
deviation). The weights are the least-squares fit of the target profile
by the snapshots’ profiles, normalized to sum to one. An empty
snapshots, or a snapshot taken after no updates or after more than
updates, raises ValueError.
ema_update
Section titled “ema_update”def ema_update(ema, params, decay)Update the EMA leaves toward params, keeping each leaf’s storage dtype.
Each leaf of ema becomes decay * average + (1 - decay) * live, and a
leaf the step left as it was keeps its average bit for bit wherever that
sum is finite, however a program fuses it (a select, not a difference
that finite leaves of opposite signs overflow; a NaN decay stays NaN);
the live value is the leaf of params at the same path; leaves that
ema does not hold are not averaged. The arithmetic runs in fp32 or
wider, so fp64 leaves keep fp64 precision. A decay of 1 or more returns
the average leaf as it was, even when it holds nonfinite values, as a
frozen reference can.
exponent
Section titled “exponent”def exponent(std: float) -> floatReturn the exponent γ of the power-function profile whose relative
standard deviation is std.
γ is the largest real root of γ³ + 7γ² + (16 - std⁻²)γ + (12 - std⁻²)
(Eq. 126 and Algorithm 2). std must lie in (0, 12^-0.5); any other
value raises ValueError.
power_decay
Section titled “power_decay”def power_decay(std: float) -> optax.ScheduleReturn the decay schedule of the power-function EMA whose relative
standard deviation is std, in the form an EMASpec reads.
After count completed updates, update count + 1 keeps
(1 - 1/(count + 1))^(γ + 1) of the average (Eq. 127). So the first
update replaces the average with the weights entirely.
The decay is computed as exp((γ + 1) log1p(-1/t)). That keeps 1 - decay, which is of order (γ + 1)/t, accurate in fp32 late in a run.
powered_decay
Section titled “powered_decay”def powered_decay(power) -> optax.ScheduleThe decay schedule of the power-function EMA whose γ + 1 is power,
a number or an array (power_decay).