Skip to content

dew.objectives.diffusion.end_to_end

REPA-E: the autoencoder trained together with the diffusion model through REPA.

REPA-E is from Leng et al. 2025, “REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformers”; the official code is End2End-Diffusion/REPA-E’s train_repae.py, models/sit.py and loss/losses.py. One step there makes three updates:

  1. The VAE, on its own loss (L1 reconstruction, LPIPS, KL and the PatchGAN’s generator term) plus vae_align_proj_coeff times the REPA loss of its latent. The diffusion model is frozen, and its batch norm reads the running statistics.
  2. The discriminator, on its hinge loss against the same reconstruction.
  3. The diffusion model, on the denoising and REPA losses of the detached latent. Its batch norm normalizes with the batch’s statistics and updates the running ones.

The three parameter sets are disjoint, and each update reads the others as they were before the step. So one loss that stops each gradient where the reference freezes it makes all three updates at once, on the same latent, times and noise.

NameSummary
AUTOENCODERWhere the trained autoencoder’s parameters sit in params.
LATENT_STATSThe collection of the latent batch norm’s running mean and variance.
PERCEPTUALThe collection that holds the frozen LPIPS network’s weights.
TUNED_WEIGHTSWhere a TunedAutoEncoder’s variables hold the trained autoencoder’s weights.
EndToEndREPA-E’s end-to-end tuning of the autoencoder.
PatchDiscriminatorpix2pix’s PatchGAN, written as taming-transformers’ NLayerDiscriminator, which REPA-E copies.
TunedAutoEncoderautoencoder normalizing its latents by a tuned run’s running statistics, read from the variables each call takes, {TUNED_WEIGHTS: weights, LATENT_STATS: statistics}: shifted by the running mean and scaled by the reciprocal square root of the running variance, without the batch norm’s epsilon, the scale REPA-E’s SiT.extract_latents_stats gives its sampling to denormalize with.

attribute source

AUTOENCODER = 'autoencoder'

Where the trained autoencoder’s parameters sit in params.

attribute source

LATENT_STATS = 'latent_statistics'

The collection of the latent batch norm’s running mean and variance.

attribute source

PERCEPTUAL = 'perceptual'

The collection that holds the frozen LPIPS network’s weights.

attribute source

TUNED_WEIGHTS = 'weights'

Where a TunedAutoEncoder’s variables hold the trained autoencoder’s weights.

dataclass source

class EndToEnd(
align_weight: float = 1.5,
reconstruction_weight: float = 1.0,
perceptual_weight: float = 1.0,
kl_weight: float = 1e-06,
discriminator_weight: float = 0.1,
discriminator_start: int = 0,
discriminator_width: int = 64,
discriminator_layers: int = 3,
momentum: float = 0.1,
epsilon: float = 0.0001,
)

REPA-E’s end-to-end tuning of the autoencoder.

align_weight is vae_align_proj_coeff, the REPA loss’s weight in the autoencoder’s update. The autoencoder’s own loss is REPA-E’s ReconstructionLoss_Single_Stage with configs/l1_lpips_kl_gan.yaml:

  • the L1 reconstruction, at reconstruction_weight;
  • LPIPS between the images and the reconstruction, at perceptual_weight (dew.eval.LPIPSNetwork, its published weights held frozen under PERCEPTUAL);
  • the posterior’s KL, at kl_weight, summed over each latent and averaged over the batch;
  • the PatchGAN’s generator term, minus the mean logit of the reconstruction, at discriminator_weight from step discriminator_start.

The PatchGAN (PatchDiscriminator at discriminator_width and discriminator_layers) trains on the hinge loss of the images against the same step’s reconstruction, in params as DISCRIMINATOR. A weight of 0 leaves its network out. An affine-free batch norm (momentum, epsilon) normalizes the latents per channel; it replaces the autoencoder’s fixed latent scale and starts from it.

discriminator: PatchDiscriminator | None

The PatchGAN the recipe trains, or None at discriminator_weight 0.

def initial_statistics(shift: ArrayLike, scale: ArrayLike, channels: int) -> Variables

Return the running statistics REPA-E’s init_bn starts from.

Per channel, the mean is the autoencoder’s latent shift and the variance is 1 / scale^2.

def tuned(
autoencoder: AutoEncoder,
variables: Variables,
) -> tuple[AutoEncoder, Variables]

Return the autoencoder a task over a tuned run’s variables decodes with, and its variables.

The trained weights come from params/autoencoder and the running statistics from LATENT_STATS, both under autoencoder (TunedAutoEncoder).

def normalized(latents: jax.Array, statistics: Variables) -> jax.Array

Return latents normalized by the running statistics, as the batch norm does in eval mode.

def batch_normalized(
latents: jax.Array,
statistics: Variables,
batch: Batch,
) -> tuple[jax.Array, Variables]

Return latents normalized by their own statistics, and the updated running statistics.

The statistics are taken over every axis but the channels, over batch’s rows (Objective.row_mean). This is torch’s BatchNorm2d in training mode, whose running variance is unbiased.

def regularizer(
images: jax.Array,
reconstruction: jax.Array,
moments: jax.Array,
*,
perceptual: Variables | None,
discriminator: Variables | None,
step: jax.Array,
batch: Batch,
) -> tuple[jax.Array, jax.Array, dict[str, jax.Array]]

Return the autoencoder’s own loss, the discriminator’s loss, and their terms.

perceptual is the LPIPS network’s variables and discriminator the PatchGAN’s, each None when its weight is 0. The generator term reads the discriminator frozen, and the discriminator’s hinge loss reads the images and the reconstruction frozen, so each loss trains only its own network, from the values the step began with. Every mean is over batch’s rows (Objective.row_mean).

Flax module source

class PatchDiscriminator(width: int = 64, layers: int = 3)

pix2pix’s PatchGAN, written as taming-transformers’ NLayerDiscriminator, which REPA-E copies.

REPA-E’s copy is loss/discriminator.py. The network is a 4x4 stride-2 convolution to width channels, then layers - 1 more stride-2 convolutions and one stride-1 convolution, each doubling the width up to 8x and batch normalized, every activation a 0.2 leaky ReLU, and a final 4x4 convolution to one logit per patch; every convolution pads by one. A fresh network draws its kernels from a normal with standard deviation 0.02, as taming’s weights_init does, while REPA-E’s recipe starts from a pretrained one (variables_from_torch). It returns the logits, [B, h, w, 1]. Its batch norms take their statistics over batch’s rows.

def variables_from_torch(state: Mapping[str, np.ndarray]) -> Variables

Return the variables of a torch NLayerDiscriminator(ndf=width, n_layers=layers) state dict.

Kernels are moved from OIHW to HWIO. The batch norms’ running statistics are not read, because no step of REPA-E reads them.

class source

class TunedAutoEncoder(autoencoder: AutoEncoder, params: Variables)

autoencoder normalizing its latents by a tuned run’s running statistics, read from the variables each call takes, {TUNED_WEIGHTS: weights, LATENT_STATS: statistics}: shifted by the running mean and scaled by the reciprocal square root of the running variance, without the batch norm’s epsilon, the scale REPA-E’s SiT.extract_latents_stats gives its sampling to denormalize with. The statistics are data, not constants of a compiled walk: one frozen autoencoder’s tuned copies are equal, so every publication of a run, each with its own statistics, samples through the program the first compiled.

def encode_batch(params, x, key=None, noise=None)
def decode_batch(params, z)
def encode_video(params, x, key=None, noise=None)
def decode_video(params, z)
def latent_shape(shape: tuple[int, ...]) -> tuple[int, ...]
def latent_affine(params, dtype) -> LatentAffine

The latent shift and scale the running statistics in params give, in dtype, which the base encode and decode apply.

def with_normalization(normalization)