dew.rl.env
Environments as two pure functions over a pytree, and the one loop over them.
An Env is reset(key) -> (state, TimeStep) and step(state, action) -> (state, TimeStep), jittable,
the shape every JAX-native simulator has (MuJoCo Playground’s, through dew.rl.playground) and the
shape a learned world model has too: a Dreamer world model is an Env whose state is its latent and
whose step is the learned dynamics (dew.objectives.rl.dreamer). So acting in a simulator and
imagining in a world model are the same env_rollout, a lax.scan of a policy and a step.
A batch of environments is Batched, jax.vmap over the state, whose leading axis shards like any
batch axis. An episode that ends restarts in place (env_rollout’s reset), as Brax’s and
Playground’s auto-reset wrappers restart it, the transition that ended it recorded first.
| Name | Summary |
|---|---|
Batched | count copies of env stepped together: jax.vmap over their states, each reset from its own key, an action and a TimeStep leading with the count axis. |
Carry | A rollout’s running state between calls: the environments’ state, the observation each acts on next, the key the next restarts and actions draw from, and the policy’s own memory (a recurrent policy’s state, None for one that keeps none). |
Driven | env (batched) stepped as an agent’s driver steps it: its observation is the whole step a world-model learner records, a dict of the observation, the reward that came with it, and whether it is the episode’s first, its last or terminal; an episode’s last observation is answered as a step of its own, and the step after it, whatever its action, starts the next episode (DreamerV3’s driver, embodied/core/driver.py). |
Env | An environment as two jittable functions over its state, a pytree: reset(key) starts an episode and step(state, action) advances it, each answering a TimeStep. |
EnvState | An environment’s state, a pytree of its own. |
Memory | A policy’s memory between steps (a recurrent policy’s state), None for one that keeps none. |
Observation | An environment’s observation: an array, or a dict of named views, each an array or a dict again. |
Params | A policy’s weights. |
Policy | policy(params, observation, key, memory) -> (action, extras, memory): an action for each environment, what to record beside it, and the policy’s memory after it (None for one that keeps none). |
TimeLimit | env whose episodes are cut after episode_length steps, each step repeating its action action_repeat times and summing their rewards (Brax’s EpisodeWrapper, which Playground’s training wraps its environments in): a cut episode is truncated, not terminated. |
TimeStep | What an environment answers a reset or a step with: the observation reached, the reward the step earned (zero at a reset), terminated where the episode ended in a terminal state, whose value is zero, and truncated where it was cut at a time limit, whose value is the state’s own. |
Transition | One step of a rollout, leading with the environment’s batch axes: the observation acted on, the action taken, what the policy reported beside it (extras, its log-probability and value), and the TimeStep that answered it, following, its observation the one the step reached before any restart. |
Tree | A pytree of arrays: an array, a dataclass of them (a flax.struct one), a dict, tuple or list of them, or nothing. |
chosen | new where mask holds, old elsewhere, leaf by leaf, mask along the leaves’ leading axes and a key leaf chosen by its data. |
driven_step | step as a world-model learner’s driver records it (Driven): its observation the dict of the observation, the reward that came with it, and whether it is its episode’s first, its last or terminal. |
env_rollout | steps steps of policy under params in env from carry, as one lax.scan: the carry after them and each step’s Transition, leading with the step axis [steps, ...]. |
start | A rollout’s carry at fresh episodes of env, the policy’s memory as it starts. |
view | An observation’s array: it whole, where name is None, or its view name. |
views | An observation’s named views: it, where it is a dict of them. |
Batched
Section titled “Batched”class Batched( env: Env = struct.field(pytree_node=False), count: int = struct.field(pytree_node=False),)count copies of env stepped together: jax.vmap over their states, each reset from its
own key, an action and a TimeStep leading with the count axis.
Batched.reset
Section titled “Batched.reset”def reset(key: jax.Array) -> tuple[EnvState, TimeStep]Batched.step
Section titled “Batched.step”def step(state: EnvState, action: jax.Array) -> tuple[EnvState, TimeStep]class Carry(memory: Memory = None)A rollout’s running state between calls: the environments’ state, the observation each
acts on next, the key the next restarts and actions draw from, and the policy’s own memory
(a recurrent policy’s state, None for one that keeps none).
Driven
Section titled “Driven”class Driven(env: Env = struct.field(pytree_node=False))env (batched) stepped as an agent’s driver steps it: its observation is the whole step a
world-model learner records, a dict of the observation, the reward that came with it, and
whether it is the episode’s first, its last or terminal; an episode’s last observation is
answered as a step of its own, and the step after it, whatever its action, starts the next
episode (DreamerV3’s driver, embodied/core/driver.py). A rollout over it passes reset=False.
Driven.reset
Section titled “Driven.reset”def reset(key: jax.Array) -> tuple[_Driven, TimeStep]Driven.step
Section titled “Driven.step”def step(state: _Driven, action: jax.Array) -> tuple[_Driven, TimeStep]class Env(Protocol)An environment as two jittable functions over its state, a pytree: reset(key) starts an
episode and step(state, action) advances it, each answering a TimeStep. action_size is the
width of a continuous action in [-1, 1].
Env.reset
Section titled “Env.reset”def reset(key: jax.Array) -> tuple[State, TimeStep]Env.step
Section titled “Env.step”def step(state: State, action: jax.Array) -> tuple[State, TimeStep]EnvState
Section titled “EnvState”An environment’s state, a pytree of its own.
Memory
Section titled “Memory”A policy’s memory between steps (a recurrent policy’s state), None for one that keeps none.
Observation
Section titled “Observation”An environment’s observation: an array, or a dict of named views, each an array or a dict again.
Params
Section titled “Params”A policy’s weights.
Policy
Section titled “Policy”policy(params, observation, key, memory) -> (action, extras, memory): an action for each
environment, what to record beside it, and the policy’s memory after it (None for one that keeps
none).
TimeLimit
Section titled “TimeLimit”class TimeLimit( env: Env = struct.field(pytree_node=False), episode_length: int = struct.field(pytree_node=False, default=1000), action_repeat: int = struct.field(pytree_node=False, default=1),)env whose episodes are cut after episode_length steps, each step repeating its action
action_repeat times and summing their rewards (Brax’s EpisodeWrapper, which Playground’s
training wraps its environments in): a cut episode is truncated, not terminated. An episode env
itself ends, terminated or truncated, ends there: within a repeat it is held as it ended, earning
nothing more.
TimeLimit.reset
Section titled “TimeLimit.reset”def reset(key: jax.Array) -> tuple[_Limited, TimeStep]TimeLimit.step
Section titled “TimeLimit.step”def step(state: _Limited, action: jax.Array) -> tuple[_Limited, TimeStep]TimeStep
Section titled “TimeStep”class TimeStep()What an environment answers a reset or a step with: the observation reached, the reward
the step earned (zero at a reset), terminated where the episode ended in a terminal state,
whose value is zero, and truncated where it was cut at a time limit, whose value is the
state’s own. Every leaf leads with the environment’s batch axes.
done: jax.Array-
Where the episode ended, by either cause.
Transition
Section titled “Transition”class Transition()One step of a rollout, leading with the environment’s batch axes: the observation acted
on, the action taken, what the policy reported beside it (extras, its log-probability and
value), and the TimeStep that answered it, following, its observation the one the step
reached before any restart.
A pytree of arrays: an array, a dataclass of them (a flax.struct one), a dict, tuple or list of
them, or nothing.
chosen
Section titled “chosen”def chosen(mask: jax.Array, new: T, old: T) -> Tnew where mask holds, old elsewhere, leaf by leaf, mask along the leaves’ leading axes and a
key leaf chosen by its data.
driven_step
Section titled “driven_step”def driven_step(step: TimeStep, first: jax.Array) -> TimeStepstep as a world-model learner’s driver records it (Driven): its observation the dict of the
observation, the reward that came with it, and whether it is its episode’s first, its last
or terminal.
env_rollout
Section titled “env_rollout”def env_rollout( env: Env, policy: Policy, params: Params, carry: Carry, steps: int, *, reset: bool = True,) -> tuple[Carry, Transition]steps steps of policy under params in env from carry, as one lax.scan: the carry
after them and each step’s Transition, leading with the step axis [steps, ...].
With reset, an environment whose episode ended restarts from a fresh reset of its own before
the next step, its transition keeping the observation the episode ended at; a learned
environment, whose imagined episodes never restart, passes False.
def start(env: Env, key: jax.Array, memory: Memory = None) -> CarryA rollout’s carry at fresh episodes of env, the policy’s memory as it starts.
def view(observation: Observation, name: str | None = None) -> jax.ArrayAn observation’s array: it whole, where name is None, or its view name.
def views(observation: Observation) -> Mapping[str, Observation]An observation’s named views: it, where it is a dict of them.