Skip to content

World models: acting and imagining in one loop

Suppose you have taught a program to balance a pole on a cart by letting it practise in a simulator, pushing the cart left and right millions of times until it stops dropping the pole. Now suppose the simulator is slow, or the real thing breaks when it falls, and every practice step costs you something. You would want the program to do some of its practising in its head, the way you might rehearse a parking manoeuvre before you try it. That is what a world model is for. It is a small, learned copy of the world, good enough to rehearse in.

Here is the thought this tutorial is built on. In Dew, an environment is two functions and nothing more. One starts an episode, the other takes a state and an action and answers with the next observation and a reward. A physics simulator fits that shape, and so does a learned model, if we let its state be a bundle of numbers it keeps in its head and its step be a guess at what comes next. Put either one in the loop below and the loop cannot tell the difference.

One loop. What the policy reads goes in, the policy's action goes to a step, and the step answers with the next thing to read and a reward. The step can be a simulator's physics, whose state is the cartpole and whose observation is five numbers, or a world model's prior, whose state is a latent and whose answer is the latent feature, h and z joined. The same scan runs both.observationwhat it readspolicyreads, choosesaction1 number herestep(state, action)physicspriorstate: the cartpole, or a latent (h, z)next readingand rewardthe policy reads 5 numbers from the simulator, or the640-wide [h, z] feature in imaginationone scan, env_rollout, whichever step

We will run that loop on MuJoCo Playground’s cartpole: first a random policy, then PPO in the simulator, then DreamerV3 practising inside its model. After that, Dreamer 4 learning from video alone, and racing drones, where the physics stays in JAX and only the camera lives outside it.

%pip install -q "dewml[cuda13,playground,hyperlux] @ git+https://github.com/AshishKumar4/dew@ff1a61a94d3b31152bd10d606204633e44534b85" --extra-index-url https://pypi.dewml.dev/simple
import time
import jax
import jax.numpy as jnp
import numpy as np
import optax
from dew import figures
from dew.rl.env import env_rollout, start
from dew.rl.playground import Playground, playground
TASK = "CartpoleBalance"
probe = Playground.load(TASK)
print(jax.devices()[0], "observation", probe.env.observation_size, "action", probe.action_size)
cuda:0 observation 5 action 1

The task is CartpoleBalance. At every step the policy is shown five numbers (we just printed the sizes above) and answers with one number between -1 and 1, how hard to push the cart and which way. An episode runs for at most 1000 steps, which is how Playground trains these tasks. Keep the pole up the whole time and the return gets close to 1000.

Before we teach anything, what does guessing get us? A random policy is the smallest policy there is. It ignores the observation completely and draws its push from its random key. It still has the shape every policy here has: weights, observation, key and memory in; action and memory out. Because it has that shape, the rollout loop can already run it, 256 carts at once.

env = playground(TASK, 256)
def random_policy(params, observation, key, memory):
action = jax.random.uniform(key, (observation.shape[0], env.action_size), minval=-1, maxval=1)
return action, {}, memory
rolled = jax.jit(lambda carry: env_rollout(env, random_policy, None, carry, 1000))
_, steps = rolled(start(env, jax.random.key(0)))
returns = np.asarray(steps.following.reward.sum(0))
print(f"random policy: mean return {returns.mean():.1f} over {len(returns)} episodes")
random policy: mean return 358.9 over 256 episodes

358.9, on average, out of about 1000. That number is the sum of rewards over each episode, averaged over the 256 carts. These episodes run to their 1000-step time limit rather than ending when the pole falls, so read it as reward collected under random pushes, not as time spent with the pole up. It gives us a floor to measure against: anything we teach from here has to beat a policy that is not even looking.

The obvious way to get better is to practise in the simulator and adjust. PPO does exactly that. It keeps two networks, an actor that says how to push and a critic that guesses how much reward is still to come from here. The critic’s guesses let it say, for each push it tried, whether things went better or worse than expected, and the actor is nudged towards the pushes that went better. The loss clips the gain from moving far from the policy that collected the batch, which discourages very large changes in one update, though nothing caps how far the weights themselves move. That clip is the one you may have met in tutorial 10. Here the actor’s push is a draw from a squashed bell curve rather than a choice among tokens, but the idea is the same.

Dew’s version follows Brax’s PPO, in big parallel batches. Below it practises on 2048 carts at once, collects a stretch of steps, and takes several passes of updates over that stretch before collecting again. At the end we score it greedily: always the centre of its bell curve, no exploring.

import itertools
from dew.data.dataset import Loader
from dew.objectives.rl.control import ActorCritic, ControlPPOObjective, ControlRollout, episode_return
from dew.training import Trainer
objective = ControlPPOObjective(ActorCritic(probe.action_size), int(probe.env.observation_size),
discount=0.995, reward_scaling=10.0, entropy_cost=1e-2)
rollout = ControlRollout(objective, playground(TASK, 2048), unroll_length=30, unrolls=16, epochs=16,
minibatches=32)
idle = Loader(train=lambda partition: itertools.repeat({"step": np.zeros((1,), np.float32)}), val=None,
records=1, batch=1)
began = time.perf_counter()
ppo = Trainer(objective, optax.adam(1e-3), key=jax.random.key(1), rollout=rollout).fit(idle, steps=8 * 512,
log_every=512)
print(f"{time.perf_counter() - began:.0f} s; greedy return",
episode_return(objective, playground(TASK, 64), ppo.variables, jax.random.key(2), 1000))
╭─  dew · ActorCritic  ───────────────────────────────────────────────────────────────────────────────────────────╮
│ 268,387 parameters  ·  on 1 × NVIDIA RTX PRO 6000 Blackwell Server Edition  ·  batch 1  ·  float32              │
│                                                                                                                 │
│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4096/4096  100%  133.9 step/s  0:00:44 │
│                                                                                                                 │
│ train                                                                                                           │
│   loss                     0.2480  █▇██▇▇▄▁     ↓ 1,352×                                                        │
│   entropy                 0.08654  ████▇▇▅▁        ↓ 86%                                                        │
│   pg_clipfrac              0.1693  ▄▁▅▁█▁▇▇       ↑ 138%                                                        │
│   policy                 -0.05414  ▇█▆█▁▇▅▁       ↓ 314%                                                        │
│   ppo_kl                  0.08489  ▃▂▄▂▆▁██       ↑ 319%                                                        │
│   value                    0.3030  █▇██▇▇▄▁     ↓ 1,107×                                                        │
│   data_wait_ms            0.01406  █▇▃▂▂▁▂▁        ↓ 14%                                                        │
│   grad_norm                 175.8  █▇▇▆▇▇▃▁        ↓ 15×                                                        │
│   peak_memory_gib          0.3771  ▁▇██████       ↑ 0.2%                                                        │
│   pg_clipfrac_lower 0   flops 6.532e+10   compiles 1.000   cache_hits 763.0   cache_misses 0                    │
│ throughput                                                                                                      │
│   step_time_ms              6.110  █▄▄▄▄▄▄▁        ↓ 51%                                                        │
│   samples_per_sec         167,592  ▁▅▅▅▅▅▅█       ↑ 103%                                                        │
│   mfu                        2.1%  ▁▅▅▅▅▅▅█     ↑ 1.1 pt                                                        │
│   hlo_flops_util             2.1%  ▁▅▅▅▅▅▅█     ↑ 1.1 pt                                                        │
│   hfu                        2.1%  ▁▅▅▅▅▅▅█     ↑ 1.1 pt                                                        │
│   rollout_seconds           3.946  █▄▁▁▁▁▁▁        ↓ 63%                                                        │
│ optimizer                                                                                                       │
│   accepted 100.0%                                                                                               │
│ rollout                                                                                                         │
│   episode_return              nan  ▁▂█                                                                          │
╰──────────────────────────────────────────────────────────────────────────────────────────────────  ✓ finished  ─╯
╭─  ✓ ActorCritic  ───────────────────────────╮
│    trained  4096 steps in 0:00:44           │
│ first step  after 9.03 s                    │
│       then  115.7 step/s                    │
│    goodput  79.7% of the wall time in steps │
│ final loss  0.2480                          │
╰─────────────────────────────────────────────╯
44 s; greedy return 999.743896484375

44 seconds, and the greedy return is 999.7. The pole simply stays up. When practising is this cheap, there is not much more to ask for. The rest of this tutorial only makes sense once practice stops being cheap.

So keep the question in mind. What would we do if each step ran on a real robot, or in a simulator too slow to call millions of times? We would want far more learning from each real step: remember every step taken, learn a model of what the world does, and practise inside the model, where steps are just arithmetic. That is DreamerV3’s plan (Hafner et al., 2023).

DreamerV3’s world model never tries to remember the cartpole in full detail. It keeps a summary, a latent state in two parts, and the split between the parts is the whole trick, so let’s go slowly.

The first part, usually written hh, is 512 numbers carried by a recurrent network, a block GRU. Think of it as the model’s running memory of how things have been going. It is updated step by step from the previous hh, the previous draw zz and the action just taken. The update itself makes no new draw: given those three, the next hh is worked out rather than sampled. Its history still runs through the earlier draws, since each step’s zz feeds the next step’s hh. The second part, zz, is where the guessing lives. It is 32 separate draws, each one picking one of a few classes, 4 in the small model we run here. If hh is what the model can work out, zz is its way of saying “and then one of several things could be true”.

Now the important move. There are two ways to get zz at a step. If we have actually seen this step’s observation, we can read zz off with the help of that observation. That is the posterior, q(zt∣ht,ot)q(z_t \mid h_t, o_t), the summary of a step we watched. If we have not seen it, because we are rehearsing a future that has not happened, we have to guess zz from hh alone. That is the prior, p(zt∣ht)p(z_t \mid h_t). Training pulls the prior towards the posterior, so that the guess made in the dark lands near the summary we would have written in the light. Where the guess can be right, imagining will track watching. Where the world is genuinely up in the air, the draws in zz give the imagination its spread.

One step of DreamerV3's latent. The deterministic state h, 512 numbers from a block GRU, carries the past. The stochastic part z is 32 categorical draws: from the posterior when the step's observation is there to read, and from the prior, h alone, when the model imagines. Training pulls the prior toward the posterior. The two together feed heads that rebuild the observation and predict reward and continuation.h (512)GRU memoryof the pastobservationthis step'sposterior: h + observation…prior: h alone…pulled close in trainingz: 32 draws, 4 classes each here[h, z] feedsrebuildrewardcontinuewatching uses the top row; imagining, the bottom one
One step of DreamerV3's latent. The deterministic state h, 512 numbers from a block GRU, carries the past. The stochastic part z is 32 categorical draws: from the posterior when the step's observation is there to read, and from the prior, h alone, when the model imagines. Training pulls the prior toward the posterior. The two together feed heads that rebuild the observation and predict reward and continuation.h (512)GRU memoryobservationthis step'sposterior: h + observation…prior: h alone…pulled close in trainingz: 32 draws, 4 classes each here[h, z] feedsrebuildrewardcontinuewatching uses the top row;imagining, the bottom one

Off the pair (h,z)(h, z) hang a few small heads. One rebuilds the observation, which forces the latent to describe the step it sits on. One predicts the reward, and one whether the episode goes on. With those heads the model can play the environment’s part: hand it a latent and an action and it advances hh, draws the next zz from the prior, and answers with a predicted reward. In Dew that is LatentEnv. The actor and critic learn by rolling out in it, starting from latents of real steps stored in the replay.

Below, the agent runs 16 carts in the real simulator to fill that replay and reuses what it stored very heavily, which is rather the point. It uses a learning rate of 10−410^{-4} where DreamerV3’s own recipe uses 4×10−54 \times 10^{-5} over far longer runs. We take the larger rate for this short run and say so.

from dew.objectives.rl.dreamer import DreamerRollout, DreamerV3Objective, WorldModel, episodes_return
from dew.optim import laprop
sys_env = playground(TASK, 16)
dreamer = DreamerV3Objective(WorldModel(int(probe.env.observation_size), probe.action_size))
replay = DreamerRollout(dreamer, sys_env, capacity=4000, batch=16, length=64, train_ratio=1024.0, prefill=64)
began = time.perf_counter()
agent = Trainer(dreamer, laprop(learning_rate=1e-4), key=jax.random.key(3), rollout=replay).fit(idle, steps=20_000,
log_every=2000)
print(f"{time.perf_counter() - began:.0f} s, {replay.environment_steps} environment steps; return",
episodes_return(dreamer, playground(TASK, 16), agent.variables, jax.random.key(4), 1000))
╭─  dew · WorldModel  ────────────────────────────────────────────────────────────────────────────────────────────╮
│ 637,446 parameters  ·  on 1 × NVIDIA RTX PRO 6000 Blackwell Server Edition  ·  batch 1  ·  float32              │
│                                                                                                                 │
│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  20000/20000  100%  32.0 step/s  0:10:37 │
│                                                                                                                 │
│ train                                                                                                           │
│   loss                        3.357  █▂▁▁▁▁▁▁▁▁        ↓ 53%                                                    │
│   imagined_return             182.5  ▁▅▆▆▆▇▇▇██       ↑ 663%                                                    │
│   loss_continue             0.02044  █▂▁▁▁▁▁▁▁▁       ↓ 0.6%                                                    │
│   loss_decoder              0.01217  █▅▆▅▅▄▄▃▁▂        ↓ 80%                                                    │
│   loss_dynamics               1.380  ▁▃▅▇█▇▇▇▆▆        ↑ 13%                                                    │
│   loss_policy             -0.006085  █▁▁▁▁▁▁▁▁▁       ↓ 101%                                                    │
│   loss_replay_value           1.123  █▄▂▂▂▂▂▁▁▁        ↓ 76%                                                    │
│   loss_representation         1.380  ▁▃▅▇█▇▇▇▆▆        ↑ 13%                                                    │
│   loss_reward                0.5109  █▄▃▂▂▁▂▂▄▂        ↓ 24%                                                    │
│   loss_value                 0.9650  █▂▂▂▂▁▂▁▁▁        ↓ 68%                                                    │
│   return_scale                106.0  ▁▃▅▆▇▇▇▇██        ↑ 14×                                                    │
│   data_wait_ms              0.01646  ▄▁▇▂▇▁▃█▁▅       ↑ 0.7%                                                    │
│   grad_norm                   3.477  █▆▅▄▃▂▂▂▁▁        ↓ 59%                                                    │
│   flops 3.288e+10   compiles 1.000   cache_hits 1,394   cache_misses 0   peak_memory_gib 0.3771                 │
│ throughput                                                                                                      │
│   step_time_ms                31.25  █▅▅▅▅▅▁▅▅▅       ↓ 5.5%                                                    │
│   samples_per_sec             512.0  ▁▄▄▄▄▄█▄▄▄       ↑ 5.8%                                                    │
│   mfu                          0.2%  ▁▄▄▄▄▄█▄▄▄            →                                                    │
│   hlo_flops_util               0.2%  ▁▄▄▄▄▄█▄▄▄            →                                                    │
│   hfu                          0.2%  ▁▄▄▄▄▄█▄▄▄            →                                                    │
│   rollout_seconds             55.34  █▃▃▃▃▃▁▃▃▃        ↓ 18%                                                    │
│ optimizer                                                                                                       │
│   accepted 100.0%                                                                                               │
│ rollout                                                                                                         │
│   episode_return                nan                                                                             │
╰──────────────────────────────────────────────────────────────────────────────────────────────────  ✓ finished  ─╯
╭─  ✓ WorldModel  ────────────────────────────╮
│    trained  20000 steps in 0:10:38          │
│ first step  after 16.30 s                   │
│       then  32.2 step/s                     │
│    goodput  97.4% of the wall time in steps │
│ final loss  3.357                           │
╰─────────────────────────────────────────────╯
638 s, 21024 environment steps; return 810.866943359375

The return is 810.9, from 21,024 real environment steps in all, and it took 638 seconds. Set against PPO’s run, those three numbers are the trade to judge.

The model is not buying you a better cartpole. PPO reached 999.7 in 44 seconds, because MuJoCo steps are nearly free and PPO spent them freely, thousands of carts at a time. Dreamer spent its 638 seconds mostly on arithmetic, replaying and imagining, and asked the simulator for only about twenty one thousand steps. On this task that trade is a bad one.

But turn the prices around in your head. Give each real step a real cost: a slow simulator, a robot that wears out, a drone that crashes. Then the same trade starts to look very different. The model’s whole value is that it converts a small pile of real steps into a large amount of practice. Whether that conversion is worth its price depends entirely on what a real step costs you, and the last part of this tutorial is about a task where steps are not free at all.

First, though, we should check the thing itself. We have been told the actor learns from imagined rewards. Do those rewards have anything to do with the real ones?

Checking the imagination against the replay

Section titled “Checking the imagination against the replay”

Here is a direct way to look. Take a stretch from the replay and watch it for 32 steps, every observation fixing the latent through the posterior. Then close your eyes: from that latent, let the prior run on for 32 more steps with the policy acting on what it imagines, and collect the predicted rewards. Beside them, put the rewards that same replayed stretch actually went on to earn. Keep in mind what this comparison can show. The imagined stretch acts with fresh draws from the policy, not with the actions stored in the replay, so the two stretches are different trajectories that share only their starting latent. Even a perfect model would not reproduce the stored rewards step for step, and matching average levels cannot rule out a model that predicts an average reward. This is a distribution-level sanity check: do the imagined rewards sit at about the right level?

Imagination. A replayed run is watched for 32 steps, each observation fixing the latent through the posterior. From there the observations stop and the prior runs on alone for 32 steps, the policy acting on imagined latents and the reward head scoring them. Three imagined continuations fan out, since each prior step draws its z.watched in the simulatorimagined in the model32 steps, posteriorobservations stop32 steps, prior and policy alonereward read from the runreward from its head

In training the same move is made constantly and more modestly: 15 imagined steps from every replayed state in a batch, and the critic learns its values from those short rehearsals. The cell below makes one 32-step rehearsal so we can compare it with what happened next.

from dew.objectives.rl.dreamer import LatentEnv
from dew.rl.distributions import twohot_bins
from dew.rl.env import Carry
windows = replay.sample(jax.random.key(5))
model = dreamer.model.bind(agent.variables, rngs={"latent": jax.random.key(6)})
deter, stoch = model.initial((16,))
first = jnp.asarray(windows["first"]).at[:, 0].set(True)
previous = jnp.concatenate([jnp.zeros_like(windows["action"][:, :1]), windows["action"][:, :-1]], 1)
for t in range(32):
deter, stoch, _ = dreamer.observe(model, deter, stoch, windows["observation"][:, t], previous[:, t], first[:, t])
world = LatentEnv(dreamer.model, agent.variables, twohot_bins())
state, feature = world.at(deter, stoch, jax.random.key(7))
def imagined_policy(params, feature, key, memory):
mean, std = dreamer.model.apply(params, feature, method=WorldModel.policy)
return jnp.clip(mean + std * jax.random.normal(key, mean.shape), -1, 1), {}, None
_, dream = env_rollout(world, imagined_policy, agent.variables, Carry(state, feature, jax.random.key(8)), 32,
reset=False)
figures.curve({"replayed": np.asarray(windows["reward"][:, 32:]).mean(0),
"imagined": np.asarray(dream.following.reward).mean(1)}, x_label="step", y_label="reward")
reward against step0.4250.450.4750102030steprewardreplayed 0.40imagined 0.44

The two lines wander in the same band and end close, the replayed mean at 0.40 and the imagined at 0.44, with the imagined line the smoother of the two in this plot. Nothing guarantees that smoothness: each imagined reward is the reward head’s mean at a freshly drawn latent under freshly drawn actions, averaged over the 16 carts, and a mean can still jump when its input does. What the check supports is the level. Over these 32 steps the imagined rewards stay near what the replay went on to earn, which is what a rehearsal needs from them. Push the imagination much further and the fan in the picture is the truth of it. Every prior step draws a fresh zz, small disagreements compound, and the rehearsal slowly becomes a story the model tells itself. Dreamer keeps its rehearsals at 15 steps for that reason.

DreamerV3’s model is built around acting. It needs actions and rewards, and its latent is one step of an agent’s life. Dreamer 4 (Hafner et al., 2025) asks a different question. What if the world model first learns from video alone, and only later has actions threaded through it? Two pieces change. A tokenizer squeezes each frame, on its own so nothing leaks backwards in time, into a small set of latents. Then a transformer over those latents learns how a sequence of frames continues.

The transformer is trained as a diffusion model, more or less. Each frame’s latents are mixed with noise up to some signal level τ\tau, different for every frame, and the model has to name the clean latents. Generating then means walking a frame from pure noise at τ=0\tau = 0 to clean at τ=1\tau = 1, and the natural worry is the one we had with samplers in tutorial 01: small safe steps take forever, and nobody wants a hundred steps per frame.

Shortcut forcing is Dreamer 4’s answer, and it is a lovely piece of self discipline. The model is told, along with τ\tau, the size dd of the step it is being asked to take. At the smallest size it simply learns to name the clean latents. At any larger size it is asked to agree with itself. Take one step of size dd, then take two of its own half steps of size d/2d/2 over the same stretch, the second starting where the first landed, and average where the two half steps point. The large step is trained to match that average. At larger sizes the target is the model’s own half steps, so training encourages the sizes to agree with each other. Agreement by itself says nothing about accuracy, as the run below shows.

Shortcut forcing. Each frame's latents sit somewhere between noise and clean, at signal level tau. Two half steps of size d over 2 cross from tau to tau plus d, the second starting where the first lands. One step of size d covers the same stretch, and training pulls its endpoint to where the two half steps land. The notebook generates each frame in four such large steps.noise, τ = 0clean, τ = 1ττ + d/2τ + dits own two half steps, d/2 eachone step, size dtraining pulls its endpoint to theirsso generation crosses in 4 large steps a frame

Our test world is as small as a world gets: two balls bounce around a 32×3232 \times 32 box for 12 frames, in 2048 clips of video, with no actions or rewards. The tokenizer squeezes each frame into 16 latents of 8 numbers and the dynamics model learns over those. At the end we show it four frames of an unseen clip and ask for the next eight, four steps a frame, beside what really happened.

def bouncing(rows, frames, size, seed):
"""Videos [rows, frames, size, size, 1] in [-1, 1] of two balls bouncing off the walls."""
rng = np.random.default_rng(seed)
grid = np.stack(np.meshgrid(np.arange(size), np.arange(size), indexing="ij"), -1) + 0.5
videos = np.full((rows, frames, size, size, 1), -1.0, np.float32)
for video in videos:
position, velocity = rng.uniform(3, size - 3, (2, 2)), rng.uniform(-1.5, 1.5, (2, 2))
for frame in video:
for ball in range(2):
frame[np.sum(np.square(grid - position[ball]), -1) <= 2.5 ** 2, 0] = 1.0
position = position + velocity
velocity = np.where((position < 2.5) | (position > size - 2.5), -velocity, velocity)
position = np.clip(position, 2.5, size - 2.5)
return videos
from dew.objectives.world_model import CausalTokenizer, Dynamics, TokenizerObjective, WorldModelObjective
videos, held = bouncing(2048, 12, 32, 0), bouncing(8, 12, 32, 1)
clips = Loader(train=lambda partition: ({"video": videos[i:i + 32]} for i in itertools.cycle(range(0, 2048, 32))),
val=None, records=2048, batch=32)
tokenizer = CausalTokenizer((32, 32, 1), width=128, depth=3, latents=16, bottleneck=8)
tokens = Trainer(TokenizerObjective(tokenizer), optax.adam(3e-4), key=jax.random.key(9)).fit(clips, steps=2000,
log_every=1000)
dynamics = Dynamics(latents=16, bottleneck=8, spatial=16, width=256, depth=6, frames=12)
world_model = WorldModelObjective(dynamics, tokenizer, tokens.variables)
trained = Trainer(world_model, optax.adam(3e-4), key=jax.random.key(10)).fit(clips, steps=3000, log_every=1000)
latents = world_model.latents(jnp.asarray(held))
predicted = world_model.generate(trained.variables, latents[:, :4], 8, jax.random.key(11), steps=4)
frames = np.asarray(tokenizer.apply(tokens.variables, predicted, method=CausalTokenizer.decode))
def pixels(video):
return np.repeat(np.round((np.clip(video, -1, 1) + 1) * 127.5).astype(np.uint8), 3, -1)
figures.images([np.concatenate([pixels(held[0, 4:]), pixels(frames[0])])], columns=8, size=52,
rows=["truth", "imagined"])
╭─  dew · CausalTokenizer  ───────────────────────────────────────────────────────────────────────────────────────╮
│ 1.6M parameters  ·  on 1 × NVIDIA RTX PRO 6000 Blackwell Server Edition  ·  batch 32  ·  float32                │
│                                                                                                                 │
│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  2000/2000  100%  42.7 step/s  0:00:51 │
│                                                                                                                 │
│ train                                                                                                           │
│   loss                   0.1465  ▁█       ↑ 0.2%                                                                │
│   mse                    0.1465  ▁█       ↑ 0.2%                                                                │
│   data_wait_ms          0.01679  ▁█       ↑ 3.3%                                                                │
│   grad_norm              0.4110  █▁        ↓ 35%                                                                │
│   compiles                1.000  ▁▁            →                                                                │
│   cache_hits              1,598  ▁▁            →                                                                │
│   cache_misses                0  ▁▁            →                                                                │
│   peak_memory_gib         2.516  ▁█            →                                                                │
│ throughput                                                                                                      │
│   step_time_ms            23.41  █▁       ↓ 6.1%                                                                │
│   samples_per_sec         1,367  ▁█       ↑ 6.5%                                                                │
│   hlo_flops_util           2.7%  ▁█     ↑ 0.2 pt                                                                │
│   hfu                      2.7%  ▁█     ↑ 0.2 pt                                                                │
│ optimizer                                                                                                       │
│   accepted               100.0%  ▁▁            →                                                                │
╰──────────────────────────────────────────────────────────────────────────────────────────────────  ✓ finished  ─╯
╭─  ✓ CausalTokenizer  ───────────────────────╮
│    trained  2000 steps in 0:00:51           │
│ first step  after 4.49 s                    │
│       then  42.7 step/s                     │
│    goodput  91.2% of the wall time in steps │
│ final loss  0.1465                          │
╰─────────────────────────────────────────────╯
╭─  dew · Dynamics  ──────────────────────────────────────────────────────────────────────────────────────────────╮
│ 6.3M parameters  ·  on 1 × NVIDIA RTX PRO 6000 Blackwell Server Edition  ·  batch 32  ·  float32                │
│                                                                                                                 │
│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  3000/3000  100%  44.1 step/s  0:01:13 │
│                                                                                                                 │
│ train                                                                                                           │
│   loss                  0.01494  █▁▁        ↓ 74%                                                               │
│   frame_loss            0.01494  █▁▁        ↓ 74%                                                               │
│   data_wait_ms          0.01695  ▁█▄       ↑ 0.8%                                                               │
│   grad_norm              0.9320  █▂▁        ↓ 84%                                                               │
│   compiles 1.000   cache_hits 1,600   cache_misses 0   peak_memory_gib 2.516                                    │
│ throughput                                                                                                      │
│   step_time_ms            22.65  █▁▁       ↓ 8.3%                                                               │
│   samples_per_sec         1,413  ▁██       ↑ 9.1%                                                               │
│   hlo_flops_util           4.7%  ▁██     ↑ 0.4 pt                                                               │
│   hfu                      4.7%  ▁██     ↑ 0.4 pt                                                               │
│ optimizer                                                                                                       │
│   accepted 100.0%                                                                                               │
╰──────────────────────────────────────────────────────────────────────────────────────────────────  ✓ finished  ─╯
╭─  ✓ Dynamics  ──────────────────────────────╮
│    trained  3000 steps in 0:01:14           │
│ first step  after 5.77 s                    │
│       then  44.1 step/s                     │
│    goodput  92.2% of the wall time in steps │
│ final loss  0.01494                         │
╰─────────────────────────────────────────────╯
16 imagestruthimagined

Look at the two rows. The top row is the truth, two pale balls drifting a little across the eight frames. The bottom row is the model’s continuation, decoded back into pictures, and there are no balls in it to follow, just a faint wash where they should be. At this size and after these few thousand steps, the visible continuation is not there yet.

The numbers hint at where it stands. The dynamics loss fell by about three quarters over its 3000 steps, to 0.01494. That loss is the shortcut-forcing objective, whose larger step sizes are scored against the model’s own half-step predictions rather than the true next latents, so the fall says the model meets its training objective better. It does not by itself say the next frame is forecast well, and the decoded row above is the forecast test. The tokenizer, after 2000 steps, is still at 0.1465, and whatever the dynamics has learned does not yet survive the trip back through it into pictures you can read. Dreamer 4’s own results come from far larger models trained far longer on far richer video, and its dynamics model can also be given the actions taken between frames, which is what would make a model like this one somewhere an agent could act. What this little run gives you is the machinery and the training signal, short of a watchable continuation. The acting half, where the model really does stay in charge of the future, is where the drones come in.

A first-person racing drone is an awkward case for everything so far. Its physics is ordinary arithmetic: rotors, drag, motor lag, a battery that sags. It fits the two-function shape, and thousands of drones can fly inside one compiled rollout. Its camera is another matter. Drawing what the drone sees means ray tracing gates, pillars, ground and sky, and that work does not live inside JAX. So Dew splits the race in two along that seam.

First-person gate racing, split in two. On one side, the quadrotor physics is a JAX environment, so 4096 drones fly inside one jitted rollout and a teacher learns from the privileged state, the next gates' corners and the obstacles. On the other, a ray tracer outside JAX draws each drone's camera, stepped from the host a batch at a time; the racing policy sees that image alone, the next gate lit. The camera picture shown is a frame recorded in the tutorial.in JAX: an EnvFlightEnvrotors, drag, lag,battery sag4096 at onceteacher readsstate, gate corners,obstaclesoutside JAX: a SimulatorGateRacedrawn by a ray tracercamera image,next gate litpolicy readsthe image aloneactions cross; only the drawing stays outside
First-person gate racing, split in two. On one side, the quadrotor physics is a JAX environment, so 4096 drones fly inside one jitted rollout and a teacher learns from the privileged state, the next gates' corners and the obstacles. On the other, a ray tracer outside JAX draws each drone's camera, stepped from the host a batch at a time; the racing policy sees that image alone, the next gate lit. The camera picture shown is a frame recorded in the tutorial.in JAX: an EnvFlightEnvrotors, drag, lag, battery sag4096 at onceteacher readsstate, gate corners, obstaclessame raceoutside JAX: a SimulatorGateRacedrawn by a ray tracercamera image,next gate litpolicy readsthe image alone

On the JAX side sits FlightEnv. Every episode gets a course of its own: a loop or a chain of gates with pillars scattered among them. It starts somewhere on that course. Every drone is also drawn a little differently, in its thrust, its rate limits, its command latency and the tilt of its camera, the way real drones differ. On the host side sits GateRace, which steps the same races and asks a renderer, here the HyperLux ray tracer over a handful of CC0 textures, to draw each camera. A thing stepped from the host like that is a Simulator rather than an Env, but it answers the same questions, so rollouts and scoring treat the two alike. Only the drawing lives outside.

Notice what each side may see, because the next tutorial hangs on it. The flying policy gets the camera image alone, with the next gate lit. The drone’s full state, the next gates’ corners and the pillars are computed anyway and shown to no policy. They are privileged. A teacher may read them, and a world model may be asked to predict them. In a moment we will do the first of those.

!curl -sLO https://raw.githubusercontent.com/AshishKumar4/dew/ff1a61a94d3b31152bd10d606204633e44534b85/tools/fpv_assets.py
!python fpv_assets.py fpv-assets
from dew.rl.flight import GateRace, Racing
from dew.rl.hyperlux import HyperLuxRenderer
racing = Racing()
race = GateRace(HyperLuxRenderer(16, "fpv-assets"), racing, views=("image", "passed"))
shown = [np.asarray(race.reset(0)["image"])]
for _ in range(60):
stepped = race.step(jnp.zeros((16, 4)))
shown.append(np.asarray(stepped.observation["image"]))
figures.images(shown, columns=8, size=64, captions=["start", "after a second of hovering"])
{
 "license": "CC0 (polyhaven.com/license) textures; geometry Dew's",
 "ground": [
  "ground_aerial_grass_rock.glb",
  "ground_forrest_ground_01.glb",
  "ground_aerial_asphalt_01.glb",
  "ground_aerial_beach_01.glb",
  "ground_rocky_terrain_02.glb",
  "ground_brown_mud_leaves_01.glb"
 ],
 "sky": [
  "sky_kloofendal_48d_partly_cloudy_puresky.glb",
  "sky_lilienstein.glb",
  "sky_the_sky_is_on_fire.glb",
  "sky_kloppenheim_02.glb"
 ],
 "gate": [
  "gate_orange.glb",
  "gate_blue.glb",
  "gate_white.glb",
  "gate_padded.glb"
 ],
 "lit": [
  "lit_green.glb",
  "lit_chevrons.glb",
  "lit_magenta.glb",
  "lit_cyan.glb"
 ],
 "pillar": [
  [
   "pillar_6_concrete_floor_worn_001.glb",
   "pillar_6_rust_coarse_01.glb"
  ],
  [
   "pillar_12_concrete_floor_worn_001.glb",
   "pillar_12_rust_coarse_01.glb"
  ],
  [
   "pillar_24_concrete_floor_worn_001.glb",
   "pillar_24_rust_coarse_01.glb"
  ]
 ],
 "prop": [
  "prop_tree.glb",
  "prop_rock.glb",
  "prop_banner.glb"
 ],
 "drone": "drone.glb",
 "aspects": [
  6.0,
  12.0,
  24.0
 ],
 "gate_geometry": {
  "opening": 1.5,
  "frame": 0.25,
  "depth": 0.1
 }
}
16 images at each of 2 pointsstartafter a second of hovering

Those are 16 drones at their starts and again after 60 steps under a zero action, about a second at this race’s 60 steps a second. A zero action asks for hover thrust and no body rates. It is a neutral command, not a command to hold position, and a drone that started already flying keeps its motion under it. Every view is different because every start and every camera is drawn afresh. This, at far lower resolution, is all a racing policy ever gets to see.

It raises an obvious question. If we want good flying to learn from, why make the first learner peer through a camera? Let one policy read the privileged numbers directly: how the drone moves and turns, the next two gates’ corners, the nearest obstacles, even its own airframe’s drawn physics, packed into one observation of width TEACHER. It flies 4096 drones at once inside JAX, no images anywhere, so its practice is cheap as PPO’s was. What it learns could later be shown to a camera policy, or fill a world model’s replay; that longer programme lives in tools/fpv_race.py. Here we only ask whether the teacher itself can learn the race.

To keep its first lesson gentle, the course is loosened up. Only loop courses, and both the course difficulty and the drone randomization turned down to a fifth. We score the teacher before and after training on 256 fresh drones of its own arrangement, counting the gates each drone passes in its best episode over the 25 seconds an episode lasts, and how often a drone crashes along the way.

from dew.rl.env import Batched
from dew.rl.flight import TEACHER, FlightEnv
easy = Racing(families=(("loop", 1.0),))
teacher = ControlPPOObjective(ActorCritic(4, (256, 256, 256), (256, 256, 256, 256), spread=0.2), TEACHER,
discount=0.99, clip=0.2, entropy_cost=1e-4, reads=("teacher",))
flights = ControlRollout(teacher, Batched(FlightEnv(easy, ("teacher",), difficulty=0.2, ramp=0.2), 4096),
unroll_length=64, epochs=4, minibatches=32)
began = time.perf_counter()
taught = Trainer(teacher, optax.adam(3e-4), key=jax.random.key(12), rollout=flights).fit(idle, steps=25_600,
log_every=2560)
print(f"{time.perf_counter() - began:.0f} s of training")
judge = Batched(FlightEnv(easy, ("teacher", "passed"), difficulty=0.2, ramp=0.2), 256)
flown = jax.jit(lambda params, carry: env_rollout(judge, teacher.greedy, params, carry, easy.max_steps))
for name, params in (("fresh", teacher.init(jax.random.key(13))), ("trained", taught.variables)):
_, steps = flown(params, start(judge, jax.random.key(14)))
passed = np.asarray(steps.following.observation["passed"]).max(0)
crashes = np.asarray(steps.following.terminated).sum(0)
print(f"{name} teacher over {easy.episode_seconds:.0f} s: {passed.mean():.1f} gates in its best episode, "
f"{crashes.mean():.1f} crashes a drone")
╭─  dew · ActorCritic  ───────────────────────────────────────────────────────────────────────────────────────────╮
│ 370,697 parameters  ·  on 1 × NVIDIA RTX PRO 6000 Blackwell Server Edition  ·  batch 1  ·  float32              │
│                                                                                                                 │
│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  25600/25600  100%  131.9 step/s  0:03:25 │
│                                                                                                                 │
│ train                                                                                                           │
│   loss                      1.330  ▁▃▄█▁▅▂▆▂█       ↑ 324%                                                      │
│   entropy                  -5.531  █▆▆▅▅▃▃▂▁▁        ↓ 70%                                                      │
│   pg_clipfrac             0.09082  ▁▁▄▂▇▆▄▅█▄        ↑ 60%                                                      │
│   policy                 0.002198  ▁▄▁▂▃▄▁▃█▃       ↑ 307%                                                      │
│   ppo_kl                  0.01384  ▃▁▂▁▆▆▄▇█▄        ↑ 32%                                                      │
│   value                     1.327  ▁▃▄█▁▅▂▆▂█       ↑ 322%                                                      │
│   data_wait_ms            0.01431  ▄█▃▁▆▃▂▄▁▁       ↓ 4.9%                                                      │
│   grad_norm                 15.20  ▁▄▆█▇▆▆▇▇█       ↑ 896%                                                      │
│   pg_clipfrac_lower 0   flops 2.104e+10   compiles 1.000   cache_hits 2,752   cache_misses 0   peak_memory_gib  │
│ 2.516                                                                                                           │
│ throughput                                                                                                      │
│   step_time_ms              7.665  █▄▄▃▃▃▃▃▄▁       ↓ 4.5%                                                      │
│   samples_per_sec          16,698  ▁▅▅▆▆▆▆▆▅█       ↑ 4.7%                                                      │
│   mfu                        0.5%  ▁▅▅▆▆▆▆▆▅█            →                                                      │
│   hlo_flops_util             0.5%  ▁▅▅▆▆▆▆▆▅█            →                                                      │
│   hfu                        0.5%  ▁▅▅▆▆▆▆▆▅█            →                                                      │
│   rollout_seconds           17.23  █▁▁▁▁▁▁▁▁▁        ↓ 23%                                                      │
│ optimizer                                                                                                       │
│   accepted 100.0%                                                                                               │
│ rollout                                                                                                         │
│   episode_return            457.2  ▁▁▁▂▄▅▆▆▇█            ↑                                                      │
╰──────────────────────────────────────────────────────────────────────────────────────────────────  ✓ finished  ─╯
╭─  ✓ ActorCritic  ───────────────────────────╮
│    trained  25600 steps in 0:03:25          │
│ first step  after 5.62 s                    │
│       then  128.4 step/s                    │
│    goodput  97.3% of the wall time in steps │
│ final loss  1.330                           │
╰─────────────────────────────────────────────╯
205 s of training
fresh teacher over 25 s: 0.0 gates in its best episode, 12.3 crashes a drone
trained teacher over 25 s: 34.6 gates in its best episode, 0.2 crashes a drone

Before training, no gates at all and 12.3 crashes a drone, which is what random body rates do to a quadrotor. After 205 seconds of training, 34.6 gates in a best episode and 0.2 crashes a drone. The teacher has plainly learned to race its easy loops, and it learned the way PPO learned the cartpole, by spending cheap steps without counting them.

What it has not done is learn to see. Everything it knows arrived as numbers no flying drone will ever be handed. A camera pilot has to get the same knowledge out of a small image, and a world model is well shaped for that: train its latent to predict the privileged numbers from the image, so its summary of a flight must contain where the drone and the gate are, and let the policy fly on that summary. Tutorial 14 does that, from scratch and with no teacher at all, on SkyDreamer’s race.