dew.objectives.diffusion.end_to_end
REPA-E: the autoencoder trained together with the diffusion model through REPA.
REPA-E is from Leng et al. 2025, “REPA-E: Unlocking VAE for End-to-End Tuning
with Latent Diffusion Transformers”; the official code is
End2End-Diffusion/REPA-E’s train_repae.py, models/sit.py and
loss/losses.py. One step there makes three updates:
- The VAE, on its own loss (L1 reconstruction, LPIPS, KL and the PatchGAN’s
generator term) plus
vae_align_proj_coefftimes the REPA loss of its latent. The diffusion model is frozen, and its batch norm reads the running statistics. - The discriminator, on its hinge loss against the same reconstruction.
- The diffusion model, on the denoising and REPA losses of the detached latent. Its batch norm normalizes with the batch’s statistics and updates the running ones.
The three parameter sets are disjoint, and each update reads the others as they were before the step. So one loss that stops each gradient where the reference freezes it makes all three updates at once, on the same latent, times and noise.
| Name | Summary |
|---|---|
AUTOENCODER | Where the trained autoencoder’s parameters sit in params. |
LATENT_STATS | The collection of the latent batch norm’s running mean and variance. |
PERCEPTUAL | The collection that holds the frozen LPIPS network’s weights. |
TUNED_WEIGHTS | Where a TunedAutoEncoder’s variables hold the trained autoencoder’s weights. |
EndToEnd | REPA-E’s end-to-end tuning of the autoencoder. |
PatchDiscriminator | pix2pix’s PatchGAN, written as taming-transformers’ NLayerDiscriminator, which REPA-E copies. |
TunedAutoEncoder | autoencoder normalizing its latents by a tuned run’s running statistics, read from the variables each call takes, {TUNED_WEIGHTS: weights, LATENT_STATS: statistics}: shifted by the running mean and scaled by the reciprocal square root of the running variance, without the batch norm’s epsilon, the scale REPA-E’s SiT.extract_latents_stats gives its sampling to denormalize with. |
AUTOENCODER
Section titled “AUTOENCODER”AUTOENCODER = 'autoencoder'Where the trained autoencoder’s parameters sit in params.
LATENT_STATS
Section titled “LATENT_STATS”LATENT_STATS = 'latent_statistics'The collection of the latent batch norm’s running mean and variance.
PERCEPTUAL
Section titled “PERCEPTUAL”PERCEPTUAL = 'perceptual'The collection that holds the frozen LPIPS network’s weights.
TUNED_WEIGHTS
Section titled “TUNED_WEIGHTS”TUNED_WEIGHTS = 'weights'Where a TunedAutoEncoder’s variables hold the trained autoencoder’s weights.
EndToEnd
Section titled “EndToEnd”class EndToEnd( align_weight: float = 1.5, reconstruction_weight: float = 1.0, perceptual_weight: float = 1.0, kl_weight: float = 1e-06, discriminator_weight: float = 0.1, discriminator_start: int = 0, discriminator_width: int = 64, discriminator_layers: int = 3, momentum: float = 0.1, epsilon: float = 0.0001,)REPA-E’s end-to-end tuning of the autoencoder.
align_weight is vae_align_proj_coeff, the REPA loss’s weight in the
autoencoder’s update. The autoencoder’s own loss is REPA-E’s
ReconstructionLoss_Single_Stage with configs/l1_lpips_kl_gan.yaml:
- the L1 reconstruction, at
reconstruction_weight; - LPIPS between the images and the reconstruction, at
perceptual_weight(dew.eval.LPIPSNetwork, its published weights held frozen underPERCEPTUAL); - the posterior’s KL, at
kl_weight, summed over each latent and averaged over the batch; - the PatchGAN’s generator term, minus the mean logit of the reconstruction,
at
discriminator_weightfrom stepdiscriminator_start.
The PatchGAN (PatchDiscriminator at discriminator_width and
discriminator_layers) trains on the hinge loss of the images against the
same step’s reconstruction, in params as DISCRIMINATOR. A weight of 0
leaves its network out. An affine-free batch norm (momentum, epsilon)
normalizes the latents per channel; it replaces the autoencoder’s fixed
latent scale and starts from it.
discriminator: PatchDiscriminator | None-
The PatchGAN the recipe trains, or None at
discriminator_weight0.
EndToEnd.initial_statistics
Section titled “EndToEnd.initial_statistics”def initial_statistics(shift: ArrayLike, scale: ArrayLike, channels: int) -> VariablesReturn the running statistics REPA-E’s init_bn starts from.
Per channel, the mean is the autoencoder’s latent shift and the variance is 1 / scale^2.
EndToEnd.tuned
Section titled “EndToEnd.tuned”def tuned( autoencoder: AutoEncoder, variables: Variables,) -> tuple[AutoEncoder, Variables]Return the autoencoder a task over a tuned run’s variables decodes with, and its variables.
The trained weights come from params/autoencoder and the running
statistics from LATENT_STATS, both under autoencoder (TunedAutoEncoder).
EndToEnd.normalized
Section titled “EndToEnd.normalized”def normalized(latents: jax.Array, statistics: Variables) -> jax.ArrayReturn latents normalized by the running statistics, as the batch norm does in eval mode.
EndToEnd.batch_normalized
Section titled “EndToEnd.batch_normalized”def batch_normalized( latents: jax.Array, statistics: Variables, batch: Batch,) -> tuple[jax.Array, Variables]Return latents normalized by their own statistics, and the updated running statistics.
The statistics are taken over every axis but the channels, over batch’s
rows (Objective.row_mean). This is torch’s BatchNorm2d in training mode,
whose running variance is unbiased.
EndToEnd.regularizer
Section titled “EndToEnd.regularizer”def regularizer( images: jax.Array, reconstruction: jax.Array, moments: jax.Array, *, perceptual: Variables | None, discriminator: Variables | None, step: jax.Array, batch: Batch,) -> tuple[jax.Array, jax.Array, dict[str, jax.Array]]Return the autoencoder’s own loss, the discriminator’s loss, and their terms.
perceptual is the LPIPS network’s variables and discriminator the
PatchGAN’s, each None when its weight is 0. The generator term reads the
discriminator frozen, and the discriminator’s hinge loss reads the images and
the reconstruction frozen, so each loss trains only its own network, from
the values the step began with. Every mean is over batch’s rows
(Objective.row_mean).
PatchDiscriminator
Section titled “PatchDiscriminator”class PatchDiscriminator(width: int = 64, layers: int = 3)pix2pix’s PatchGAN, written as taming-transformers’ NLayerDiscriminator, which REPA-E copies.
REPA-E’s copy is loss/discriminator.py. The network is a 4x4 stride-2
convolution to width channels, then layers - 1 more stride-2 convolutions
and one stride-1 convolution, each doubling the width up to 8x and batch
normalized, every activation a 0.2 leaky ReLU, and a final 4x4 convolution to
one logit per patch; every convolution pads by one. A fresh network draws its
kernels from a normal with standard deviation 0.02, as taming’s
weights_init does, while REPA-E’s recipe starts from a pretrained one
(variables_from_torch). It returns the logits, [B, h, w, 1]. Its batch
norms take their statistics over batch’s rows.
PatchDiscriminator.variables_from_torch
Section titled “PatchDiscriminator.variables_from_torch”def variables_from_torch(state: Mapping[str, np.ndarray]) -> VariablesReturn the variables of a torch NLayerDiscriminator(ndf=width, n_layers=layers) state dict.
Kernels are moved from OIHW to HWIO. The batch norms’ running statistics are not read, because no step of REPA-E reads them.
TunedAutoEncoder
Section titled “TunedAutoEncoder”class TunedAutoEncoder(autoencoder: AutoEncoder, params: Variables)autoencoder normalizing its latents by a tuned run’s running statistics, read from the
variables each call takes, {TUNED_WEIGHTS: weights, LATENT_STATS: statistics}: shifted by the
running mean and scaled by the reciprocal square root of the running variance, without the batch
norm’s epsilon, the scale REPA-E’s SiT.extract_latents_stats gives its sampling to denormalize
with. The statistics are data, not constants of a compiled walk: one frozen autoencoder’s tuned
copies are equal, so every publication of a run, each with its own statistics, samples through
the program the first compiled.
TunedAutoEncoder.encode_batch
Section titled “TunedAutoEncoder.encode_batch”def encode_batch(params, x, key=None, noise=None)TunedAutoEncoder.decode_batch
Section titled “TunedAutoEncoder.decode_batch”def decode_batch(params, z)TunedAutoEncoder.encode_video
Section titled “TunedAutoEncoder.encode_video”def encode_video(params, x, key=None, noise=None)TunedAutoEncoder.decode_video
Section titled “TunedAutoEncoder.decode_video”def decode_video(params, z)TunedAutoEncoder.latent_shape
Section titled “TunedAutoEncoder.latent_shape”def latent_shape(shape: tuple[int, ...]) -> tuple[int, ...]TunedAutoEncoder.latent_affine
Section titled “TunedAutoEncoder.latent_affine”def latent_affine(params, dtype) -> LatentAffineThe latent shift and scale the running statistics in params give, in dtype, which the
base encode and decode apply.
TunedAutoEncoder.with_normalization
Section titled “TunedAutoEncoder.with_normalization”def with_normalization(normalization)