Skip to content

dew.rl.env

Environments as two pure functions over a pytree, and the one loop over them.

An Env is reset(key) -> (state, TimeStep) and step(state, action) -> (state, TimeStep), jittable, the shape every JAX-native simulator has (MuJoCo Playground’s, through dew.rl.playground) and the shape a learned world model has too: a Dreamer world model is an Env whose state is its latent and whose step is the learned dynamics (dew.objectives.rl.dreamer). So acting in a simulator and imagining in a world model are the same env_rollout, a lax.scan of a policy and a step.

A batch of environments is Batched, jax.vmap over the state, whose leading axis shards like any batch axis. An episode that ends restarts in place (env_rollout’s reset), as Brax’s and Playground’s auto-reset wrappers restart it, the transition that ended it recorded first.

NameSummary
Batchedcount copies of env stepped together: jax.vmap over their states, each reset from its own key, an action and a TimeStep leading with the count axis.
CarryA rollout’s running state between calls: the environments’ state, the observation each acts on next, the key the next restarts and actions draw from, and the policy’s own memory (a recurrent policy’s state, None for one that keeps none).
Drivenenv (batched) stepped as an agent’s driver steps it: its observation is the whole step a world-model learner records, a dict of the observation, the reward that came with it, and whether it is the episode’s first, its last or terminal; an episode’s last observation is answered as a step of its own, and the step after it, whatever its action, starts the next episode (DreamerV3’s driver, embodied/core/driver.py).
EnvAn environment as two jittable functions over its state, a pytree: reset(key) starts an episode and step(state, action) advances it, each answering a TimeStep.
EnvStateAn environment’s state, a pytree of its own.
MemoryA policy’s memory between steps (a recurrent policy’s state), None for one that keeps none.
ObservationAn environment’s observation: an array, or a dict of named views, each an array or a dict again.
ParamsA policy’s weights.
Policypolicy(params, observation, key, memory) -> (action, extras, memory): an action for each environment, what to record beside it, and the policy’s memory after it (None for one that keeps none).
TimeLimitenv whose episodes are cut after episode_length steps, each step repeating its action action_repeat times and summing their rewards (Brax’s EpisodeWrapper, which Playground’s training wraps its environments in): a cut episode is truncated, not terminated.
TimeStepWhat an environment answers a reset or a step with: the observation reached, the reward the step earned (zero at a reset), terminated where the episode ended in a terminal state, whose value is zero, and truncated where it was cut at a time limit, whose value is the state’s own.
TransitionOne step of a rollout, leading with the environment’s batch axes: the observation acted on, the action taken, what the policy reported beside it (extras, its log-probability and value), and the TimeStep that answered it, following, its observation the one the step reached before any restart.
TreeA pytree of arrays: an array, a dataclass of them (a flax.struct one), a dict, tuple or list of them, or nothing.
chosennew where mask holds, old elsewhere, leaf by leaf, mask along the leaves’ leading axes and a key leaf chosen by its data.
driven_stepstep as a world-model learner’s driver records it (Driven): its observation the dict of the observation, the reward that came with it, and whether it is its episode’s first, its last or terminal.
env_rolloutsteps steps of policy under params in env from carry, as one lax.scan: the carry after them and each step’s Transition, leading with the step axis [steps, ...].
startA rollout’s carry at fresh episodes of env, the policy’s memory as it starts.
viewAn observation’s array: it whole, where name is None, or its view name.
viewsAn observation’s named views: it, where it is a dict of them.

dataclass source

class Batched(
env: Env = struct.field(pytree_node=False),
count: int = struct.field(pytree_node=False),
)

count copies of env stepped together: jax.vmap over their states, each reset from its own key, an action and a TimeStep leading with the count axis.

def reset(key: jax.Array) -> tuple[EnvState, TimeStep]
def step(state: EnvState, action: jax.Array) -> tuple[EnvState, TimeStep]

dataclass source

class Carry(memory: Memory = None)

A rollout’s running state between calls: the environments’ state, the observation each acts on next, the key the next restarts and actions draw from, and the policy’s own memory (a recurrent policy’s state, None for one that keeps none).

dataclass source

class Driven(env: Env = struct.field(pytree_node=False))

env (batched) stepped as an agent’s driver steps it: its observation is the whole step a world-model learner records, a dict of the observation, the reward that came with it, and whether it is the episode’s first, its last or terminal; an episode’s last observation is answered as a step of its own, and the step after it, whatever its action, starts the next episode (DreamerV3’s driver, embodied/core/driver.py). A rollout over it passes reset=False.

def reset(key: jax.Array) -> tuple[_Driven, TimeStep]
def step(state: _Driven, action: jax.Array) -> tuple[_Driven, TimeStep]

class source

class Env(Protocol)

An environment as two jittable functions over its state, a pytree: reset(key) starts an episode and step(state, action) advances it, each answering a TimeStep. action_size is the width of a continuous action in [-1, 1].

def reset(key: jax.Array) -> tuple[State, TimeStep]
def step(state: State, action: jax.Array) -> tuple[State, TimeStep]

attribute source

An environment’s state, a pytree of its own.

attribute source

A policy’s memory between steps (a recurrent policy’s state), None for one that keeps none.

attribute source

An environment’s observation: an array, or a dict of named views, each an array or a dict again.

attribute source

A policy’s weights.

attribute source

policy(params, observation, key, memory) -> (action, extras, memory): an action for each environment, what to record beside it, and the policy’s memory after it (None for one that keeps none).

dataclass source

class TimeLimit(
env: Env = struct.field(pytree_node=False),
episode_length: int = struct.field(pytree_node=False, default=1000),
action_repeat: int = struct.field(pytree_node=False, default=1),
)

env whose episodes are cut after episode_length steps, each step repeating its action action_repeat times and summing their rewards (Brax’s EpisodeWrapper, which Playground’s training wraps its environments in): a cut episode is truncated, not terminated. An episode env itself ends, terminated or truncated, ends there: within a repeat it is held as it ended, earning nothing more.

def reset(key: jax.Array) -> tuple[_Limited, TimeStep]
def step(state: _Limited, action: jax.Array) -> tuple[_Limited, TimeStep]

dataclass source

class TimeStep()

What an environment answers a reset or a step with: the observation reached, the reward the step earned (zero at a reset), terminated where the episode ended in a terminal state, whose value is zero, and truncated where it was cut at a time limit, whose value is the state’s own. Every leaf leads with the environment’s batch axes.

done: jax.Array

Where the episode ended, by either cause.

dataclass source

class Transition()

One step of a rollout, leading with the environment’s batch axes: the observation acted on, the action taken, what the policy reported beside it (extras, its log-probability and value), and the TimeStep that answered it, following, its observation the one the step reached before any restart.

attribute source

A pytree of arrays: an array, a dataclass of them (a flax.struct one), a dict, tuple or list of them, or nothing.

function source

def chosen(mask: jax.Array, new: T, old: T) -> T

new where mask holds, old elsewhere, leaf by leaf, mask along the leaves’ leading axes and a key leaf chosen by its data.

function source

def driven_step(step: TimeStep, first: jax.Array) -> TimeStep

step as a world-model learner’s driver records it (Driven): its observation the dict of the observation, the reward that came with it, and whether it is its episode’s first, its last or terminal.

function source

def env_rollout(
env: Env,
policy: Policy,
params: Params,
carry: Carry,
steps: int,
*,
reset: bool = True,
) -> tuple[Carry, Transition]

steps steps of policy under params in env from carry, as one lax.scan: the carry after them and each step’s Transition, leading with the step axis [steps, ...].

With reset, an environment whose episode ended restarts from a fresh reset of its own before the next step, its transition keeping the observation the episode ended at; a learned environment, whose imagined episodes never restart, passes False.

function source

def start(env: Env, key: jax.Array, memory: Memory = None) -> Carry

A rollout’s carry at fresh episodes of env, the policy’s memory as it starts.

function source

def view(observation: Observation, name: str | None = None) -> jax.Array

An observation’s array: it whole, where name is None, or its view name.

function source

def views(observation: Observation) -> Mapping[str, Observation]

An observation’s named views: it, where it is a dict of them.