Skip to content

dew.nn.autoencoders

Autoencoders a latent diffusion model reads and writes its samples through: the interface (AutoEncoder) and the KL and Stable Diffusion VAEs. Each name is imported from its own module when first read, so a reader of the interface alone reads neither VAE.

NameSummary
AutoEncoderEncodes images or video to the latents a latent diffusion model trains on, and decodes them back.
AutoencoderKLEncodes NHWC images to latents with a convolutional VAE, and decodes latents back to images.
LatentAffineThe affine map an autoencoder’s latents are normalized by, as the arrays a trace holds (AutoEncoder.latent_affine): (z - shift) * factor and back as z / factor + shift, or, one that divides, (z - shift) / factor and back as z * factor + shift.
LatentNormalizationA latent normalization, one description: the source values an autoencoder’s record keeps (record, as given) beside the shift and factor its autoencoder derived from them, written together once and read-only after.
StableDiffusionVAEEncodes and decodes images with a frozen native AutoencoderKL and its latent normalization.

class source

class AutoEncoder(ABC)

Encodes images or video to the latents a latent diffusion model trains on, and decodes them back.

A subclass encodes and decodes one batch of frames, [B, H, W, C] to [B, h, w, c] and back. encode and decode here take video [B, T, H, W, C] frame by frame, unless the subclass encodes time itself (encode_video, decode_video, latent_shape), and they apply the latent normalization (normalization, LatentNormalization). On the way out, latents are normalized as (z - latent_shift) * latent_scale, and on the way in the normalization is inverted, the SD3 convention. latent_shift and latent_scale are each one number or one per latent channel, read from normalization; one that divides by its factor (Qwen- Image’s) rounds as a division does. The default is the identity; an autoencoder normalized by the dataset’s own latent mean and 1/std, so the diffusion model sees roughly unit-variance, zero-mean inputs, is with_normalization’s.

The weights are an argument, as a ConditionEncoder’s are. params holds what a run loaded, and every call takes the tree to use, so the layout places the weights and the checkpoint stores them.

downscale_factor: int

H / h, the spatial factor between a frame and its latent.

stochastic: bool

Whether encoding samples its latent, so a training step draws each example’s noise (encode_batch). A deterministic encoder (DC-AE, RAE) is False and refuses noise, raising TypeError naming itself.

latent_channels: int

c, the number of channels in a latent.

def towers() -> tuple[nn.Module, ...]

The networks this autoencoder runs, whose quantization the programs that call it compile for (dew.nn.kernels.options.compiler_options): its model, none for one that computes without a network (a pooling one).

def record_arguments() -> dict

The subclass’s constructor arguments besides model and params, as it was given them (a built-in’s compute dtype and latent normalization), which its record rebuilds it with (dew.registry.to_record). A subclass a run records states them.

def with_normalization(normalization: LatentNormalization) -> Self

This autoencoder normalizing its latents by normalization instead: a copy, this one unchanged.

def recorded_fields() -> dict

The record a run keeps of this autoencoder, without its parameters (dew.registry.Recorded): its module, whose record names its class, and record_arguments, an array as a list; bound passes them back to its constructor with params.

def bound(record, *, params: Variables) -> AutoEncoder

Rebuild the autoencoder a run recorded (dew.registry.to_record), around the checkpoint’s own params, reading no source: its class found as every component’s is (dew.registry.component), of Dew’s or of a package the run trusts, and its fields read against its constructor.

def encode_batch(
params,
x: jnp.ndarray,
key: jax.Array | None = None,
noise: jnp.ndarray | None = None,
) -> jnp.ndarray

Encode frames [B, H, W, C] to raw latents [B, h, w, c].

key draws a stochastic encoder’s sample, and None takes its mean. noise, a standard normal draw of the latents’ shape, is the sample’s in place of one from key (posterior_latent); a deterministic encoder reads neither.

def decode_batch(params, z: jnp.ndarray) -> jnp.ndarray

Decode raw latents [B, h, w, c] to frames [B, H, W, C].

def moments(params, images: jnp.ndarray) -> jnp.ndarray

Return the KL posterior of frames [B, H, W, C], its mean and log-variance stacked on channels.

A KL autoencoder’s latent is one draw from it normalized by latent_shift and latent_scale alone, encode being (posterior_latent(moments, key) - latent_shift) * latent_scale, so a loss may train it through the posterior and renormalize the draw (REPA-E). Any other autoencoder raises TypeError, naming itself.

def decode_raw(params, latents: jnp.ndarray) -> jnp.ndarray

Decode a draw from moments’ posterior to frames; raises as moments does.

def latent_shape(shape: tuple[int, ...]) -> tuple[int, ...]

Return the latent shape for one example of shape.

shape is (H, W, C) or, for video, (T, H, W, C).

def encode_video(
params,
x: jnp.ndarray,
key: jax.Array | None = None,
noise: jnp.ndarray | None = None,
) -> jnp.ndarray

Encode video [B, T, H, W, C] to raw latents, frame by frame.

def decode_video(params, z: jnp.ndarray) -> jnp.ndarray

Decode raw latents [B, t, h, w, c] to video, frame by frame.

def latent_affine(params, dtype) -> LatentAffine

The map encode normalizes latents by and decode restores them by, in dtype, for the weights params: this autoencoder’s configured normalization (normalization), whatever the weights. A subclass whose normalization is data the variables hold reads it from params.

def encode(
params,
x: jnp.ndarray,
key: jax.Array | None = None,
noise: jnp.ndarray | None = None,
) -> jnp.ndarray

Encode images [B, H, W, C] or video [B, T, H, W, C] to normalized latents, a stochastic encoder’s sample drawn from key or read off noise (encode_batch), by latent_affine.

def decode(params, z: jnp.ndarray) -> jnp.ndarray

Decode normalized latents [B, h, w, c] or [B, t, h, w, c] back to images or video, by latent_affine.

Flax module source

class AutoencoderKL(
channels: tuple[int, ...] = (128, 256, 512, 512),
latent_channels: int = 4,
image_channels: int = 3,
blocks_per_level: int = 2,
norm_groups: int = 32,
quantize: bool = True,
post_quantize: bool = True,
decoder_channels: tuple[int, ...] | None = None,
dtype: Dtype = jnp.float32,
)

Encodes NHWC images to latents with a convolutional VAE, and decodes latents back to images.

encode(image, key=None) returns the posterior mean, and passing a key samples its diagonal Gaussian. decode(latents) returns unnormalized image pixels. The latent scale and shift belong to AutoEncoder.

decoder_channels: tuple[int, ...] | None

The decoder’s widths where they differ from the encoder’s, as a distilled decoder’s do; None uses channels.

downscale_factor: int

The spatial factor between an image and its latent.

Every encoder level except the last halves each spatial axis, so the factor is 2 ** (len(channels) - 1).

def from_diffusers(config: Mapping[str, object], dtype: Dtype) -> AutoencoderKL

The model a diffusers AutoencoderKL config describes, computing in dtype. The stack computes SiLU, attention in the bottleneck, a decoder writing the image’s channels and the plain encoder and decoder levels; a config asking for anything else is refused by name.

def setup()
def moments(image)

Return the posterior’s mean and log-variance, stacked on the channel axis.

def encode(image, key=None, noise=None)
def decode(latents)

dataclass source

class LatentAffine(divides: bool = struct.field(pytree_node=False, default=False))

The affine map an autoencoder’s latents are normalized by, as the arrays a trace holds (AutoEncoder.latent_affine): (z - shift) * factor and back as z / factor + shift, or, one that divides, (z - shift) / factor and back as z * factor + shift. A configured normalization gives its host values (LatentNormalization.affine); a tuned run, the running statistics each call’s variables hold. divides is static, so the map crosses a jit boundary as its two arrays.

def normalized(latent: jnp.ndarray) -> jnp.ndarray
def restored(z: jnp.ndarray) -> jnp.ndarray

dataclass source

class LatentNormalization(
record: Mapping[str, RecordValue],
shift: float | np.ndarray,
factor: float | np.ndarray,
divides: bool = False,
)

A latent normalization, one description: the source values an autoencoder’s record keeps (record, as given) beside the shift and factor its autoencoder derived from them, written together once and read-only after. Latents are normalized as (z - shift) * factor and restored as z / factor + shift; one that divides normalizes as (z - shift) / factor and restores as z * factor + shift, as its source rounds.

scale: float | np.ndarray

The factor a normalized latent is the shifted one times.

def affine(dtype) -> LatentAffine

The map this normalization applies, its shift and factor in dtype.

class source

class StableDiffusionVAE(
modelname='CompVis/stable-diffusion-v1-4',
revision='bf16',
dtype=jnp.bfloat16,
latent_shift=None,
latent_scale=None,
params=None,
model: AutoencoderKL | None = None,
)

Encodes and decodes images with a frozen native AutoencoderKL and its latent normalization.

def record_arguments() -> dict
def moments(params, images)

Return AutoencoderKL.moments of the frames, which encode_batch draws its latent from.

def decode_raw(params, latents)

Decode a posterior draw as decode_batch does, traced in the caller’s step, not jitted apart.