dew.nn.mp
EDM2’s magnitude-preserving layers.
The layers follow Karras et al. 2024, “Analyzing and Improving the Training
Dynamics of Diffusion Models”, section 3 and appendix B. Every layer keeps
the expected magnitude of its activations at one. The weights are normalized
in the forward pass and scaled by the fan-in (Eq. 47), the nonlinearity is
divided by its expected output magnitude (Eq. 81), and a sum or a
concatenation weights its operands so the result has unit magnitude
(Eqs. 88, 103). The layers are NVlabs/edm2’s training/networks_edm2.py, in
Flax’s channels-last layout.
Forced weight normalization (Eq. 66) also keeps each stored weight at unit
magnitude, so an update’s relative size, and with it the effective learning
rate, does not decay as the weights grow. The paper applies it in place in
the training forward pass. Here it is forced_weight_normalization, an
optimizer step that renormalizes every mp_kernel after its update;
OptimConfig.build appends it when OptimConfig.forced_weight_normalization
is set.
| Name | Summary |
|---|---|
MP_KERNEL | The parameter name of a magnitude-preserving weight. |
MPConv | Applies a convolution whose weight is normalized per output channel and scaled by the fan-in. |
MPFourier | Computes Fourier features of a scalar at unit magnitude. |
Uncertainty | Computes EDM2’s learned uncertainty u(sigma), one log-variance per example. |
forced_weight_normalization | Return an optax transformation that keeps every mp_kernel at unit magnitude. |
mp_cat | Return the channel concatenation of a and b, weighted by t and scaled to unit magnitude. |
mp_silu | Return SiLU divided by its expected output magnitude on a unit normal input. |
mp_sum | Return the interpolation (1 - t) a + t b, scaled to unit magnitude. |
normalize | Return x scaled to unit root-mean-square magnitude over axes. |
MP_KERNEL
Section titled “MP_KERNEL”MP_KERNEL = 'mp_kernel'The parameter name of a magnitude-preserving weight. Forced weight normalization keeps these weights at unit magnitude.
MPConv
Section titled “MPConv”class MPConv(kernel_size: tuple[int, ...] = ())Applies a convolution whose weight is normalized per output channel and scaled by the fan-in.
An empty kernel_size makes it a dense layer. This is Eq. 47 of the
EDM2 paper. The weight is mp_kernel, with the spatial axes first, then
the input and output channels. A convolution pads to keep the size, as
the reference’s odd kernels do, and runs as Conv’s does
(dew.nn.conv.convolution).
MPConv.spatial_shape
Section titled “MPConv.spatial_shape”def spatial_shape(shape: tuple[int, ...]) -> tuple[int, ...]MPConv.model_flops
Section titled “MPConv.model_flops”def model_flops(lengths, input_features: int) -> floatDense or convolution products; weight normalization is pointwise work.
MPFourier
Section titled “MPFourier”class MPFourier(bandwidth: float = 1.0)Computes Fourier features of a scalar at unit magnitude.
The random frequencies and phases are drawn once and held in the
constants collection. This is Eq. 75 of the EDM2 paper.
Uncertainty
Section titled “Uncertainty”class Uncertainty(channels: int = 128)Computes EDM2’s learned uncertainty u(sigma), one log-variance per example.
Magnitude-preserving Fourier features of the model time go through one
magnitude-preserving dense layer. This is Eq. 21 of the EDM2 paper, and
the logvar head of the reference’s Precond.
forced_weight_normalization
Section titled “forced_weight_normalization”def forced_weight_normalization() -> optax.GradientTransformationReturn an optax transformation that keeps every mp_kernel at unit magnitude.
It changes each mp_kernel update so that the kernel has unit magnitude
per output channel once the update is applied (Eq. 66 of the EDM2
paper). Every other update passes through unchanged. The transformation
needs the parameters and raises ValueError without them.
mp_cat
Section titled “mp_cat”def mp_cat(a: jax.Array, b: jax.Array, t: float = 0.5) -> jax.ArrayReturn the channel concatenation of a and b, weighted by t and scaled to unit magnitude.
This is Eq. 103 of the EDM2 paper.
mp_silu
Section titled “mp_silu”def mp_silu(x: jax.Array) -> jax.ArrayReturn SiLU divided by its expected output magnitude on a unit normal input.
This is Eq. 81 of the EDM2 paper.
mp_sum
Section titled “mp_sum”def mp_sum(a: jax.Array, b: jax.Array, t: float = 0.5) -> jax.ArrayReturn the interpolation (1 - t) a + t b, scaled to unit magnitude.
This is Eq. 88 of the EDM2 paper.
normalize
Section titled “normalize”def normalize( x: jax.Array, axes: Sequence[int] | None = None, eps: float = 0.0001, *, compensated: bool = False,) -> jax.ArrayReturn x scaled to unit root-mean-square magnitude over axes.
axes defaults to all but the last, which normalizes each output channel
of a kernel, or each feature vector. compensated sums the squares with
compensated_sum, as every kernel’s normalization does (drawn_kernel,
MPConv, forced_weight_normalization): over a kernel’s fan-in a plain
float32 sum left the kernels EDM2 trains 1.5 times as far from float64 as
EDM2’s own (tests/test_edm2.py).