dew.objectives.diffusion.alignment
Representation alignment: REPA and iREPA.
REPA (Yu et al. 2025, “Representation Alignment for Generation: Training
Diffusion Transformers Is Easier Than You Think”) adds a term to the
denoising loss: the negative cosine similarity between a frozen encoder’s
patch features of the clean image and a projection of the denoiser’s hidden
tokens at one layer, averaged over tokens and examples. iREPA (Singh et al.
2026, “What matters for Representation Alignment: Global Information or
Spatial Structure?”) projects with one convolution over the token grid where
REPA uses an MLP, and z-scores the encoder’s features over space. The official code is
sihyun-yu/REPA’s loss.py and models/sit.py, and End2End-Diffusion/iREPA’s
projectors.py and spatial_zscore.
| Name | Summary |
|---|---|
ALIGNMENT | The projector’s key in params, beside the model’s own modules. |
REPRESENTATION | The variables collection that holds the frozen encoder’s weights, as encoders hold theirs. |
Alignment | Aligns the model’s hidden tokens at layer with a frozen encoder’s features. |
Projector | Projects hidden tokens to features with REPA’s build_mlp or iREPA’s convolution. |
spatial_zscore | Return iREPA’s spatial normalization of [B, N, D] features. |
ALIGNMENT
Section titled “ALIGNMENT”ALIGNMENT = 'alignment_projector'The projector’s key in params, beside the model’s own modules.
REPRESENTATION
Section titled “REPRESENTATION”REPRESENTATION = 'representation'The variables collection that holds the frozen encoder’s weights, as encoders hold theirs.
Alignment
Section titled “Alignment”class Alignment( layer: str = 'dit_block_7', encoder: nn.Module | None = None, source: str | None = 'facebook/dinov2-base', weight: float = 0.5, projector: Literal['mlp', 'conv'] = 'mlp', width: int = 2048, kernel_size: int = 3, spatial_norm: float | None = None, resolution: int | None = 224, mean: tuple[float, float, float] = IMAGENET_MEAN, std: tuple[float, float, float] = IMAGENET_STD,)Aligns the model’s hidden tokens at layer with a frozen encoder’s features.
encoder is a Flax module that maps pixels [B, S, S, 3] to patch
features [B, N, D] or [B, h, w, D], any module a record names, as a
model is named. Its weights are the starting tree’s REPRESENTATION
when it holds them, else source’s: a transformers Dinov2Model
checkpoint (repo, repo@revision or a directory), by default REPA’s
DINOv2-B/14, which a run’s record pins to a commit. With encoder None
the encoder is the source’s own, read at resolution pixels. The pixels
are resized to resolution (None keeps the data’s size) and normalized
by mean and std. layer names the submodule of the model whose
output is aligned (dit_block_7, REPA’s depth 8 on simple_dit). That
output must be a [B, N, width] token sequence in raster order over the
same grid as the encoder’s patches.
weight is REPA’s proj_coeff. projector="mlp" with width is REPA’s
projector. projector="conv" with kernel_size and a spatial_norm
gamma (0.6 in iREPA’s training script) is iREPA’s, which z-scores each
feature over the tokens after subtracting gamma times its spatial mean.
The pixel statistics default to ImageNet’s, which DINOv2 uses; REPA’s
DINOv2 reads 224 pixels of a 256 image. The loss adds weight / 2 times
the alignment term, because Dew’s L2 halves the denoising error that
REPA adds the term to.
Alignment.network
Section titled “Alignment.network”def network(held: Variables | None) -> tuple[Alignment, Variables]This alignment with its encoder, and the encoder’s weights: held, a
starting tree’s REPRESENTATION, when given, else source’s.
Alignment.targets
Section titled “Alignment.targets”def targets(variables: Variables, images: jax.Array) -> jax.ArrayReturn the encoder’s [B, N, D] features of images in [-1, 1], with no gradient.
Under iREPA (spatial_norm set) the features are spatially normalized.
Alignment.module
Section titled “Alignment.module”def module(features: int) -> ProjectorAlignment.init
Section titled “Alignment.init”def init(key, width: int, count: int, features: int) -> VariablesReturn the projector’s variables for count hidden tokens of width.
Alignment.loss
Section titled “Alignment.loss”def loss( projector: Variables, hidden: jax.Array, targets: jax.Array, batch: Batch,) -> jax.ArrayReturn REPA’s projection loss, the mean of -cos(target, projection) over tokens and examples.
The examples are batch’s rows (Objective.row_mean).
Raises ValueError when the model’s token count differs from the
encoder’s patch count.
Alignment.captures
Section titled “Alignment.captures”def captures(module: nn.Module, method: str) -> boolReturn whether capture_intermediates should keep this call: True for layer’s output.
Projector
Section titled “Projector”class Projector(width: int = 2048, kernel_size: int = 3)Projects hidden tokens to features with REPA’s build_mlp or iREPA’s convolution.
build_mlp has two hidden SiLU layers of width. iREPA’s projector is
one same-padded convolution over the token grid, which must be square.
spatial_zscore
Section titled “spatial_zscore”def spatial_zscore(features: jax.Array, gamma: float) -> jax.ArrayReturn iREPA’s spatial normalization of [B, N, D] features.
Gamma times each feature’s mean over the tokens is subtracted, and the
result is divided by the feature’s standard deviation over the tokens
plus 1e-6. The standard deviation is torch’s unbiased Tensor.std, as
in the reference.