dew.objectives.rl.dreamer
DreamerV3 (Hafner et al., arXiv 2301.04104 v2 and Nature 2025) for vector observations, as its code computes it (danijar/dreamerv3 at e01491fad6434b2245a3b8ca201dd7faedcc458c, cited by file:line).
The world model is an RSSM: a block-diagonal GRU’s deterministic state deter (rssm.py:135-158),
a posterior over stoch categoricals of classes classes from it and the encoded observation, a
prior from it alone (1% unimix, straight-through draws, rssm.py:161-176), and heads off the two
concatenated: the symlog-MSE decoder, the two-hot reward, the Bernoulli continue. It is trained
on replayed windows by the decoder’s, reward’s and continue’s losses and the dynamics and
representation KLs with one free nat (rssm.py:120-133, agent.py:156-182).
The world model is an Env whose state is the latent and whose step is the prior (LatentEnv), so
the actor and critic learn in imagination through the same env_rollout acting in a simulator
runs: from every replayed posterior state, 15 imagined steps (agent.py:189-201). The critic is a
two-hot head trained toward the imagination’s lambda-returns and toward its own slow EMA
(agent.py:382-447), and toward the replay’s lambda-returns at 0.3 (agent.py:449-480); the actor by
REINFORCE on the returns less the values, over the 5th-to-95th percentile range of the returns
floored at 1 (utils.py:16-72), with an entropy bonus of 3e-4.
Two differences from the code: a window starts from a zero latent at its first step, which the code’s replay context instead resumes from a latent stored at acting time; and the computation runs in float32 where the code runs bfloat16.
| Name | Summary |
|---|---|
AUXILIARY_SCALE | An auxiliary head’s loss weight, FlyDreamer’s aux_state and aux_goal (configs.yaml loss_scales). |
AgentMemory | An agent’s memory between its steps in an environment: the block GRU’s deterministic state or the TSSM’s cache, held; the last posterior draw, stoch; and the action it took, previous. |
ConvDecoder | DreamerV3’s image decoder (rssm.py:253-355, the simple decoder with bspace 8): from the deterministic state through blocks block-diagonal maps and from the categoricals through two dense layers to the smallest map, its side the image’s over 2 to the stages, summed, normed and a SiLU; then a stage for each but the last of mults, deepest first: a 2x nearest upsampling, a convolution, an RMS norm and a SiLU; a last upsampling and convolution to the image’s channels; a sigmoid, the pixels in [0, 1]. |
ConvEncoder | DreamerV3’s image encoder (rssm.py:179-248, the simple encoder): pixels [..., H, W, C] in [0, 255] scaled to [-0.5, 0.5], then a stage for each of mults: a kernel-square same-padded convolution to depth times the multiplier channels, a 2x2 max pool, an RMS norm and a SiLU; the last stage’s map flattened. |
Critic | The two-hot value head, 3 layers then zero-initialized logits (configs.yaml value). |
DreamerRollout | The trainer’s rollout of a DreamerV3Objective (dew.training.rollout.Rollout): its agent acts in env, a batched Env driven as Driven drives it or a Simulator driven alike (DrivenSimulator), writing each step to a device replay of capacity steps an environment; each update trains on batch windows of length replayed steps while the agent takes environment steps so that each replays train_ratio steps (the code’s run.train_ratio), after prefill steps of each environment taken uniformly at random. |
DreamerV3Objective | DreamerV3 over a WorldModel (the module docstring). |
LatentEnv | The world model as an Env: its state the latent and a key, its step the GRU and a draw from the prior, its observation the latent’s feature, its reward the reward head’s two-hot mean and its terminated never set: the continue head’s probability weighs an imagined step instead (agent.py:396-405). |
WorldModel | DreamerV3’s networks over observations of observation_size and continuous actions of action_size in [-1, 1]. |
demonstration_replay | Demonstrations as a replay to pretrain from, one stream of every episode in order, in the convention the agent records its own steps (Driven): an episode’s row t holds the observation t, the reward that came with it (none at its first), and the action taken at it, after it was seen (FlyDreamer’s recorder stored each action one row later, with the observation it produced, and so shifted every action-conditioned prediction). |
episodes_return | The mean undiscounted return of the first episode of each of env’s environments, an Env or a Simulator, over steps steps of the agent’s draws from fresh episodes: the whole episode where steps is its length (dew.rl.evaluation.episodes, the environment driven for the agent). |
lambda_return | The code’s lambda-returns [B, T - 1] of sequences [B, T] (agent.py:482-490): from the last bootstrap backward, each step’s reward plus the discounted blend of its bootstrap and the next return, cut where the episode ended (last) or terminated (term). |
AUXILIARY_SCALE
Section titled “AUXILIARY_SCALE”AUXILIARY_SCALE = 0.5An auxiliary head’s loss weight, FlyDreamer’s aux_state and aux_goal (configs.yaml loss_scales).
AgentMemory
Section titled “AgentMemory”class AgentMemory()An agent’s memory between its steps in an environment: the block GRU’s deterministic state or the
TSSM’s cache, held; the last posterior draw, stoch; and the action it took, previous.
ConvDecoder
Section titled “ConvDecoder”class ConvDecoder( depth: int = 64, units: int = 1024, mults: tuple[int, ...] = (2, 3, 4, 4), kernel: int = 5, blocks: int = 8, dtype: Dtype = jnp.float32,)DreamerV3’s image decoder (rssm.py:253-355, the simple decoder with bspace 8): from the
deterministic state through blocks block-diagonal maps and from the categoricals through two
dense layers to the smallest map, its side the image’s over 2 to the stages, summed, normed and
a SiLU; then a stage for each but the last of mults, deepest first: a 2x nearest upsampling, a
convolution, an RMS norm and a SiLU; a last upsampling and convolution to the image’s channels;
a sigmoid, the pixels in [0, 1].
ConvEncoder
Section titled “ConvEncoder”class ConvEncoder( depth: int = 64, mults: tuple[int, ...] = (2, 3, 4, 4), kernel: int = 5, dtype: Dtype = jnp.float32,)DreamerV3’s image encoder (rssm.py:179-248, the simple encoder): pixels [..., H, W, C] in
[0, 255] scaled to [-0.5, 0.5], then a stage for each of mults: a kernel-square same-padded
convolution to depth times the multiplier channels, a 2x2 max pool, an RMS norm and a SiLU; the
last stage’s map flattened. The defaults are configs.yaml’s enc.simple at its 200M size.
Critic
Section titled “Critic”class Critic(dtype: Dtype = jnp.float32)The two-hot value head, 3 layers then zero-initialized logits (configs.yaml value).
DreamerRollout
Section titled “DreamerRollout”class DreamerRollout( objective: DreamerV3Objective, env: Env | Simulator, capacity: int = 100000, batch: int = 16, length: int = 64, train_ratio: float = 1024.0, prefill: int = 64, replay: ReplayBuffer | None = None, checkpointed_replay: bool = False, teacher: Teacher | None = None, demonstrator: Teacher | None = None,)The trainer’s rollout of a DreamerV3Objective (dew.training.rollout.Rollout): its agent acts
in env, a batched Env driven as Driven drives it or a Simulator driven alike
(DrivenSimulator), writing each step to a device replay of capacity steps an environment; each
update trains on batch windows of length replayed steps while the agent takes environment steps
so that each replays train_ratio steps (the code’s run.train_ratio), after prefill steps of
each environment taken uniformly at random. A replay given (demonstrations recorded in the
agent’s convention from as many environments, [capacity, envs, ...]) is the replay it starts from
and writes on, in place of the prefill. metrics reports the mean return of the episodes that ended
since the last update. Given a privileged teacher (dew.rl.aggregation.Teacher), a policy of the
environment’s observation, every step the rollout takes, the prefill’s too, is written with the
teacher’s action at it, its label (dew.rl.aggregation.labelled), which an objective that imitates
it learns: DAgger, the replay aggregating the student’s steps beside the teacher’s own flights it
was given, labelled alike. Given a demonstrator (a Teacher too, a policy of the environment’s
observation, such as a teacher flying its policy’s draws, DART), it flies the prefill in place of the
uniform actions, so the replay starts from its demonstrations, labelled where there is a teacher; the
agent’s memory passes through its steps untouched, as through the uniform ones. The prefill is flown
length steps at a time, so a long prefill of images holds one window’s flight beside the replay, not
its whole.
The run’s checkpoints hold the environment steps it has taken (saved), so a resumed run takes the
steps it still owes; its environments begin again and its replay is made again, from replay or the
prefill. A rollout owes each update from the step it goes on from at its own pace, so a run resumed
with another batch, length or train ratio (a later phase’s longer windows) takes the steps of its
updates from there at that pace alone. Resumed from a step that holds none of its state (one an
earlier Dew saved), it owes the steps of the updates after that step alone. A cooldown branch runs on
copies of its environments and replay (branch), a Simulator keeping its own
(dew.rl.simulator.branched). With checkpointed_replay, its checkpoints also hold its replay, its
environments mid episode and their running returns, and a resumed run samples what the uninterrupted
run did, at the replay’s size a checkpoint (a JAX Env’s alone: a Simulator’s environments live
outside JAX).
environment_steps: int-
The steps every environment has taken so far, summed.
held_replay: ReplayBuffer | None-
The replay the rollout writes and samples, None before its first call.
DreamerRollout.saved
Section titled “DreamerRollout.saved”def saved() -> Saved | NoneWhat the run’s checkpoints hold of the rollout (Rollout’s saved): the environment steps it
has taken and those its prefills took, and with checkpointed_replay its replay, its environments
mid episode and their running returns; None before its first call.
DreamerRollout.saved_shapes
Section titled “DreamerRollout.saved_shapes”def saved_shapes() -> Mapping[str, object]saved’s leaves’ shapes and dtypes, before any call.
DreamerRollout.restored
Section titled “DreamerRollout.restored”def restored(tree: Saved) -> NoneThe rollout carrying on from a saved tree: owing the steps after those it had taken, and with
checkpointed_replay from its replay and its environments as they stood.
DreamerRollout.branch
Section titled “DreamerRollout.branch”def branch(name: str) -> Iterator[None]A cooldown branch’s calls (Rollout’s branch), on copies of the rollout’s environments and
replay, after which the rollout stands as the run left it: its live state is donated to the steps
that advance it, so the run’s is copied aside first, at the replay’s size again for the branch; a
begun Simulator keeps its own environments (DrivenSimulator.branch).
DreamerRollout.sample
Section titled “DreamerRollout.sample”def sample(key: jax.Array) -> Batchbatch windows of length steps drawn from the replay as an update draws them, fetched to the
host.
DreamerV3Objective
Section titled “DreamerV3Objective”class DreamerV3Objective( model: WorldModel, *, horizon: float = 333.0, imagination: int = 15, lam: float = 0.95, entropy: float = 0.0003, free_nats: float = 1.0, slow_decay: float = 0.98, return_rate: float = 0.01, variables: Variables | None = None, scales: Mapping[str, float] | None = None, imitation: float = 0.0, imitates: str = 'action', slow_targets: bool = False, smoothness: float = 0.0,)DreamerV3 over a WorldModel (the module docstring). A batch holds replayed windows [B, T]
of DreamerV3’s steps (Driven): observation, the action taken at it, the reward that came
with it, and whether it was its episode’s first, last or terminal. The scales weigh the
losses (configs.yaml loss_scales); horizon sets the continue target’s discount, 1 - 1/333;
imagination is the imagined steps from each replayed state, lam the lambda-returns’ lambda,
entropy the actor’s bonus, slow_decay the slow critic’s EMA and return_rate the rate of
the returns’ percentile EMA. The slow critic is the EMA slot over the critic alone; with
slow_targets the imagined returns bootstrap from it and the advantage is taken over it (configs.yaml
slowtar, agent.py’s tarval, as SkyDreamer sets it). A decoder scale of 0 rebuilds no observation:
an informed world model’s auxiliary heads decode privileged entries alone (SkyDreamer, Sec II-B).
smoothness weighs the squared change of the policy’s mean between imagined steps, added to the
policy’s loss (SkyDreamer, arXiv 2510.14783 Eq 11, its 0.002: on the mean alone, so the entropy bonus
is untouched).
resumable-
The entropy bonus weighs how the policy explores, which SkyDreamer lowers for its last phase; the horizon, lambda and the rest say what the critic and world model estimate.
DreamerV3Objective.held_variables
Section titled “DreamerV3Objective.held_variables”def held_variables() -> Variables | NoneDreamerV3Objective.fresh_variables
Section titled “DreamerV3Objective.fresh_variables”def fresh_variables(key: jax.Array, held: Variables | None) -> VariablesDreamerV3Objective.observe
Section titled “DreamerV3Objective.observe”def observe( model: WorldModel, deter: jax.Array, stoch: jax.Array, observation: Observation, previous: jax.Array, first: jax.Array,) -> tuple[jax.Array, jax.Array, jax.Array]One posterior step: the latent and action masked where an episode starts, the GRU, then the posterior’s draw; the new latent, beside the posterior’s log-probabilities.
DreamerV3Objective.observed
Section titled “DreamerV3Objective.observed”def observed( variables: Variables, observation: Observation, previous: jax.Array, first: jax.Array, key: jax.Array,) -> tuple[jax.Array, jax.Array, jax.Array, Window | None]The posterior latents over windows [B, T, ...] from a zero latent: each step’s
deterministic state, drawn categoricals and posterior log-probabilities, each step drawing from
its own of T keys split from key, as the code’s scan seeds each step’s sample
(rssm.py:63-90); and a transformer’s window, whose keys and values imagination from each step
attends to (TSSM.window), None for the recurrent core.
DreamerV3Objective.act
Section titled “DreamerV3Objective.act”def act( variables: Params, observed: Observation, key: jax.Array, memory: Memory,) -> tuple[jax.Array, Tree, AgentMemory]The agent’s step in an environment (dew.rl.env.Policy over Driven): the posterior
given the step’s observation, then an action drawn from the policy, clipped to [-1, 1].
DreamerV3Objective.greedy
Section titled “DreamerV3Objective.greedy”def greedy( variables: Params, observed: Observation, key: jax.Array, memory: Memory,) -> tuple[jax.Array, Tree, AgentMemory]act’s step with the policy’s mean in place of its draw, as a control objective’s greedy
scores a policy: the latent is still the posterior’s draw, as the world model reads it.
DreamerV3Objective.memory
Section titled “DreamerV3Objective.memory”def memory(count: int) -> AgentMemoryThe agent’s memory at the start for each of count environments: a zero latent, or a TSSM’s
empty cache of its context steps, and a zero previous latent and action.
DreamerV3Objective.model_flops
Section titled “DreamerV3Objective.model_flops”def model_flops(model, batch: Batch) -> floatOne update’s forward work on batch, its loss’s traced program’s (traced_update_flops).
DreamerV3Objective.training_flops
Section titled “DreamerV3Objective.training_flops”def training_flops(model, batch: Batch) -> floatOne update’s work on batch, its loss’s gradient’s traced program’s (traced_update_flops).
DreamerV3Objective.loss
Section titled “DreamerV3Objective.loss”def loss(variables: Variables, batch: Batch, step: Step) -> tuple[Ratio, Aux]LatentEnv
Section titled “LatentEnv”class LatentEnv(model: WorldModel = struct.field(pytree_node=False))The world model as an Env: its state the latent and a key, its step the GRU and a draw from
the prior, its observation the latent’s feature, its reward the reward head’s two-hot mean and
its terminated never set: the continue head’s probability weighs an imagined step instead
(agent.py:396-405). It starts where it is given (at), never by reset.
LatentEnv.at
Section titled “LatentEnv.at”def at( deter: jax.Array, stoch: jax.Array, key: jax.Array, cache: Cache | None = None,) -> tuple[_Latent, jax.Array]The latent imagination starts from, with the cache a TSSM’s next step attends to.
LatentEnv.reset
Section titled “LatentEnv.reset”def reset(key: jax.Array) -> tuple[_Latent, TimeStep]LatentEnv.step
Section titled “LatentEnv.step”def step(state: _Latent, action: jax.Array) -> tuple[_Latent, TimeStep]WorldModel
Section titled “WorldModel”class WorldModel( image: tuple[int, int, int] | None = None, conv_depth: int = 64, reads: str | None = None, proprio: tuple[tuple[str, int], ...] = (), auxiliary: tuple[tuple[str, int, int], ...] = (), dynamics: TSSM | None = None, deter: int = 512, hidden: int = 64, stoch: int = 32, classes: int = 4, blocks: int = 8, units: int = 64, bins: int = 255, unimix: float = 0.01, min_std: float = 0.1, max_std: float = 1.0, dtype: Dtype = jnp.float32,)DreamerV3’s networks over observations of observation_size and continuous actions of
action_size in [-1, 1]. The defaults are the code’s 1M model (configs.yaml size1m, which its
DM Control proprio preset trains): a 512-wide deterministic state in 8 blocks, 32 categoricals of
4 classes, 64 hidden and 64 units a layer.
image: tuple[int, int, int] | None-
An image observation’s shape
(H, W, C), uint8 pixels, its sides divisible by 16: encoded and decoded by convolutions (ConvEncoder,ConvDecoder,conv_depthchannels); None for a vector ofobservation_size. reads: str | None-
Where observations are dicts, the entry the encoder reads and the decoder rebuilds (FlyDreamer’s
image); the others are privileged, read by theauxiliaryheads alone. proprio: tuple[tuple[str, int], ...]-
Vector entries the encoder reads beside the image, each
(entry, width), symlogged, concatenated and through an MLP beside the convolutions (DreamerV3’s encoder overmlp_keys, nets.py; SkyDreamer’s body rates, motor speeds and flight plan, arXiv 2510.14783 Fig 3): sensed, so the policy reads them; decoding them is the auxiliary heads’. auxiliary: tuple[tuple[str, int, int], ...]-
Privileged entries the latent learns to predict, each
(entry, width, units): two layers ofunitsthen a linear map to the entry’swidth, its squared error summed (FlyDreamer’saux_state_headandaux_goal_head, dreamerv3/agent.py:72-78 and :207-217, configs.yaml{layers: 2, output: mse}). dynamics: TSSM | None-
TransDreamer’s transformer dynamics (
dew.objectives.rl.tssm.TSSM) in place of the block GRU, its own posterior and prior; None for the GRU’s (RSSM). dtype: Dtype-
The networks’ computation dtype, bfloat16 for speed (DreamerV3’s COMPUTE_DTYPE): parameters stay at least float32, and norms, distributions, heads’ outputs and losses at least float32. A TSSM’s dynamics compute in its own dtype.
WorldModel.setup
Section titled “WorldModel.setup”def setup() -> NoneWorldModel.example
Section titled “WorldModel.example”def example(batch: int) -> jax.Array | dict[str, jax.Array]batch zero observations as the encoder reads them (seen): a vector, an image, or an image
beside its proprioceptive entries.
WorldModel.initial
Section titled “WorldModel.initial”def initial(batch: tuple[int, ...]) -> tuple[jax.Array, jax.Array]WorldModel.seen
Section titled “WorldModel.seen”def seen(observation: Observation) -> jax.Array | dict[str, jax.Array]What the encoder reads of an observation: its reads view, or the observation whole; beside an
image, its proprio entries concatenated (example’s structure).
WorldModel.predicted
Section titled “WorldModel.predicted”def predicted(name: str, feature: jax.Array) -> jax.ArrayThe auxiliary head name’s prediction of its privileged entry.
WorldModel.embed
Section titled “WorldModel.embed”def embed(observation: Observation) -> jax.ArrayThe encoder’s tokens of an observation (seen): a vector’s symlogged first (rssm.py:218), an
image’s pixels through the convolutions, its proprioceptive entries symlogged through an MLP and
joined to them.
WorldModel.core
Section titled “WorldModel.core”def core(deter: jax.Array, stoch: jax.Array, action: jax.Array) -> jax.ArrayThe block GRU’s next deterministic state (rssm.py:135-158).
WorldModel.posterior
Section titled “WorldModel.posterior”def posterior(deter: jax.Array | None, tokens: jax.Array) -> jax.ArrayThe posterior’s log-probabilities [..., stoch, classes] (rssm.py:75-92); the TSSM’s reads
the tokens alone, the block GRU’s the deterministic state too.
WorldModel.prior
Section titled “WorldModel.prior”def prior(deter: jax.Array) -> jax.ArrayWorldModel.categorical
Section titled “WorldModel.categorical”def categorical(log_probs: jax.Array) -> jax.ArrayWorldModel.feature
Section titled “WorldModel.feature”def feature(deter: jax.Array, stoch: jax.Array) -> jax.ArrayWorldModel.decode
Section titled “WorldModel.decode”def decode(deter: jax.Array, stoch: jax.Array) -> jax.ArrayThe observation decoded from a latent: a vector’s symlogged values, an image’s pixels in [0, 1].
WorldModel.reconstruction
Section titled “WorldModel.reconstruction”def reconstruction( deter: jax.Array, stoch: jax.Array, observation: Observation,) -> jax.ArrayThe decoder’s squared error, summed over the observation: against a vector’s symlog, or an image’s pixels over 255 (rssm.py:345-355, outs.MSE summed over the image’s axes); the proprioceptive entries are the auxiliary heads’ to decode.
WorldModel.reward
Section titled “WorldModel.reward”def reward(feature: jax.Array) -> jax.ArrayWorldModel.continue_
Section titled “WorldModel.continue_”def continue_(feature: jax.Array) -> jax.ArrayWorldModel.policy
Section titled “WorldModel.policy”def policy(feature: jax.Array) -> tuple[jax.Array, jax.Array]The bounded normal’s mean, tanh of its head, and its deviation between the floor and the ceiling (heads.py:146-155).
demonstration_replay
Section titled “demonstration_replay”def demonstration_replay(episodes: Sequence[Mapping[str, np.ndarray]]) -> ReplayBufferDemonstrations as a replay to pretrain from, one stream of every episode in order, in the
convention the agent records its own steps (Driven): an episode’s row t holds the observation t,
the reward that came with it (none at its first), and the action taken at it, after it was seen
(FlyDreamer’s recorder stored each action one row later, with the observation it produced, and so
shifted every action-conditioned prediction). An episode is observation, action and reward
along its steps (an observation an array or a dict of them), and terminal where it ended in a
terminal state (only its last step can).
episodes_return
Section titled “episodes_return”def episodes_return( objective: DreamerV3Objective, env: Env | Simulator, variables: Variables, key: jax.Array, steps: int,) -> floatThe mean undiscounted return of the first episode of each of env’s environments, an Env or a
Simulator, over steps steps of the agent’s draws from fresh episodes: the whole episode where
steps is its length (dew.rl.evaluation.episodes, the environment driven for the agent).
lambda_return
Section titled “lambda_return”def lambda_return( last: jax.Array, term: jax.Array, reward: jax.Array, value: jax.Array, boot: jax.Array, discount: float, lam: float,) -> jax.ArrayThe code’s lambda-returns [B, T - 1] of sequences [B, T] (agent.py:482-490): from the last
bootstrap backward, each step’s reward plus the discounted blend of its bootstrap and the next
return, cut where the episode ended (last) or terminated (term).