Skip to content

dew.objectives.world_model

Dreamer 4’s world model at a small scale (arXiv 2509.24527, docs/research/dreamer4.md): a causal tokenizer of video frames, and an interactive dynamics transformer over its latents trained by shortcut forcing (Eqs. 4, 6-8) or, as the baseline the paper improves on, diffusion forcing.

The tokenizer turns each frame’s patches and latents learned tokens into latents x bottleneck values through a tanh, masking each patch with probability drawn from U(0, 0.9) per frame (MAE); it reads frames one at a time, so it is causal in time. The dynamics model reads each timestep’s action token, shortcut token (its signal level’s and step size’s indices) and the latents regrouped into spatial tokens, attending block-causally (a timestep’s tokens to each other and to the past), and predicts each frame’s clean latents (x-prediction).

Choices the paper leaves open (docs/research/dreamer4.md, last section) are this module’s: the widths and depths, K_max = 64, one attention over space and time in every layer, learned position embeddings, the ramp weight multiplying each frame’s loss, no LPIPS term in the tokenizer’s loss, and context frames at signal level 1 - tau_ctx when generating.

NameSummary
TOKENIZERThe collection of a world model’s tree holding its frozen tokenizer’s weights (WorldModelObjective).
CausalTokenizerA frame [..., H, W, C] in [-1, 1] to latents [..., latents, bottleneck] and back, one frame at a time (Section 3.1): patches of patch pixels and latents learned tokens through depth blocks, the latents to bottleneck channels through a tanh; the decoder’s readout tokens, one a patch, attend to themselves and the latents, the latents only to latents.
DynamicsThe interactive dynamics transformer (Section 3.2) over frames of latents x bottleneck tokenizer latents regrouped into spatial tokens, with one action token a step (a learned embedding plus a projection of action_size actions, none for unlabeled video) and one shortcut token, the embeddings of its signal level’s and step size’s grid indices at levels levels (K_max = 2 ** (levels - 1)), predicting each frame’s clean latents.
TokenizerObjectiveThe tokenizer’s MAE reconstruction (Eq.
WorldModelObjectiveTrains Dynamics over a frozen tokenizer’s latents of a batch’s video (and actions, where the model reads them) by shortcut forcing, or with shortcut False by diffusion forcing.
diffusion_forcing_lossThe baseline: each frame at its own signal level, its x-prediction regressed at the smallest step size alone, the ramp weighting it as shortcut forcing’s.
rampEq.
shortcut_forcing_lossEqs.
unit_framesA batch’s video as the objectives read it, floats in [-1, 1]: uint8 pixels, as the data workers write clips (presets.clips), are scaled to it (dew.inputs.unit_range), and floats are taken as the caller made them, already in the range.

attribute source

TOKENIZER = 'tokenizer'

The collection of a world model’s tree holding its frozen tokenizer’s weights (WorldModelObjective).

Flax module source

class CausalTokenizer(
patch: int = 4,
width: int = 64,
depth: int = 2,
heads: int = 4,
latents: int = 16,
bottleneck: int = 8,
)

A frame [..., H, W, C] in [-1, 1] to latents [..., latents, bottleneck] and back, one frame at a time (Section 3.1): patches of patch pixels and latents learned tokens through depth blocks, the latents to bottleneck channels through a tanh; the decoder’s readout tokens, one a patch, attend to themselves and the latents, the latents only to latents.

def setup() -> None
def encode(
frames: jax.Array,
mask_rate: jax.Array | None = None,
key: jax.Array | None = None,
*,
masked: jax.Array | None = None,
) -> jax.Array

Each frame’s latents; each patch masked [..., patches] replaced by the mask token, or with mask_rate [...] each with that frame’s probability, drawn from key.

def decode(latents: jax.Array) -> jax.Array

Flax module source

class Dynamics(
latents: int = 16,
bottleneck: int = 8,
spatial: int = 8,
action_size: int = 0,
width: int = 128,
depth: int = 4,
heads: int = 4,
levels: int = 7,
frames: int = 32,
)

The interactive dynamics transformer (Section 3.2) over frames of latents x bottleneck tokenizer latents regrouped into spatial tokens, with one action token a step (a learned embedding plus a projection of action_size actions, none for unlabeled video) and one shortcut token, the embeddings of its signal level’s and step size’s grid indices at levels levels (K_max = 2 ** (levels - 1)), predicting each frame’s clean latents.

class source

class TokenizerObjective(model: CausalTokenizer, *, max_mask: float = 0.9)

The tokenizer’s MAE reconstruction (Eq. 5 without LPIPS) of a batch’s video [B, T, H, W, C], each frame’s mask rate drawn from U(0, 0.9).

def fresh_variables(key: jax.Array, held: Variables | None) -> Variables
def loss(variables: Variables, batch: Batch, step: Step) -> tuple[Ratio, Aux]

class source

class WorldModelObjective(
model: Dynamics,
tokenizer: CausalTokenizer,
tokenizer_variables: Variables,
*,
shortcut: bool = True,
)

Trains Dynamics over a frozen tokenizer’s latents of a batch’s video (and actions, where the model reads them) by shortcut forcing, or with shortcut False by diffusion forcing. The tokenizer’s weights are state, under TOKENIZER beside the dynamics’ (held_variables), as a diffusion objective’s frozen towers are: they reach the compiled step as arguments, placed and checkpointed with the rest, and the optimizer never sees them.

def held_variables() -> Variables

The starting tree, or none, with the tokenizer’s weights it starts from under TOKENIZER.

def fresh_variables(key: jax.Array, held: Variables | None) -> Variables
def model_variables(variables: Variables) -> Variables

The dynamics model’s own collections out of variables: every one but the tokenizer’s.

def latents(variables: Variables, video: jax.Array) -> jax.Array

video’s latents, each frame encoded by the tokenizer variables hold (TOKENIZER).

def loss(variables: Variables, batch: Batch, step: Step) -> tuple[Ratio, Aux]
def generate(
variables: Variables,
context: jax.Array,
frames: int,
key: jax.Array,
*,
steps: int = 4,
tau_ctx: float = 0.1,
actions: jax.Array | None = None,
) -> jax.Array

frames latent frames after the context latents [B, C, ...], one at a time, each from noise in steps steps of d = 1 / steps (a shortcut model’s step size; the diffusion-forcing baseline is told the smallest), the context at signal level 1 - tau_ctx.

function source

def diffusion_forcing_loss(
f: Predictor,
clean: jax.Array,
noise: jax.Array,
tau: jax.Array,
d_min: float,
) -> jax.Array

The baseline: each frame at its own signal level, its x-prediction regressed at the smallest step size alone, the ramp weighting it as shortcut forcing’s.

function source

def ramp(tau: jax.Array) -> jax.Array

Eq. 8’s weight, 0.9 tau + 0.1.

function source

def shortcut_forcing_loss(
f: Predictor,
clean: jax.Array,
noise: jax.Array,
tau: jax.Array,
d: jax.Array,
d_min: float,
) -> jax.Array

Eqs. 6-8: each frame’s loss [B, T], its x-prediction’s squared error at the smallest step and else its velocity’s against the average of its own two half steps, held still and scaled back to x-space by (1 - tau)^2, every frame weighted by the ramp.

function source

def unit_frames(video: jax.typing.ArrayLike) -> jax.Array

A batch’s video as the objectives read it, floats in [-1, 1]: uint8 pixels, as the data workers write clips (presets.clips), are scaled to it (dew.inputs.unit_range), and floats are taken as the caller made them, already in the range.