Skip to content

dew.rl.simulator

A simulator outside JAX as an environment: the boundary any GPU simulator plugs into.

A JAX-native simulator is an Env and runs inside env_rollout’s scan. A simulator written in another framework (a CUDA, Warp, PyTorch or C++ engine) cannot step inside a jitted program, so it is stepped from the host, a batch at a time. Its arrays cross the boundary by DLPack, without a copy where both sides hold them on one device: an action is a JAX array, which a simulator takes with its own from_dlpack (torch.from_dlpack, wp.from_dlpack, cupy.from_dlpack); an observation, reward or flag is any array exposing __dlpack__, which jax.dlpack.from_dlpack takes.

A simulator steps every environment and restarts the episodes it is told to (Simulator.restart), so a driver decides when an ended episode restarts: simulator_rollout restarts it at once, recording what env_rollout records, the same Transition and Carry; DrivenSimulator restarts it a step later, as Driven drives an Env for a world-model learner. rollout_start starts either kind of environment.

A cooldown branch steps a rollout’s environments and then leaves them as the run left them (dew.training.Rollout’s branch). An Env’s are arrays the rollout keeps; a simulator keeps its own, which it puts back where it has a branch() (Branching), and one with none is refused the branch.

NameSummary
BranchingA simulator a cooldown branch steps and then leaves as the run left it: branch(), a context after which its environments stand as they did as it was entered.
DrivenSimulatorsimulator driven as Driven drives an Env, for a world-model learner: its observation is the whole step the learner records, a dict of the observation, the reward that came with it, and whether it is the episode’s first, its last or terminal; an episode’s last observation is answered as a step of its own, and the step after it, whatever its action, restarts the episode (DreamerV3’s driver, embodied/core/driver.py).
HandedAn array as a simulator hands it over: a JAX or NumPy array, or another framework’s (Shared).
SharedAn array a simulator hands over: anything exposing the DLPack protocol (a PyTorch tensor, a Warp or CuPy array), which JAX takes without a copy where both sides hold it on one device.
SimulatorA batch of environments stepped from the host (Stepped) whose step restarts none: restart(mask) starts a new episode in each environment mask [count] selects and answers every environment’s observation, the selected ones’ their new episodes’ first.
SteppedA batch of count environments stepped from the host.
branchedsimulator stepped in a cooldown branch and then left as the run left it, by its own branch() (Branching); a simulator with none is refused the branch.
receivedA simulator’s array, or a tree of them (an observation, a step, a state), each array taken as a JAX array (taken).
rollout_startA rollout’s carry at fresh episodes of env, an Env (start) or a Simulator (simulator_start).
simulator_rolloutsteps steps of policy under params in simulator from carry: the policy compiled as a program (dew.nn.kernels.options.program, once for each policy, under the options of the models it runs) and fetched once, so no step builds its dispatch key (OuterProgram.executable); the simulator stepped from the host, each step’s Transition stacked along a leading step axis [steps, ...], as env_rollout records them; the carry’s state is the simulator’s own, held by it, so the carry holds None there.
simulator_startA rollout’s carry at fresh episodes of simulator: its first observations, seeded from key, and the policy’s memory as it starts.
takenA simulator’s array as a JAX array: a device array through DLPack, on the device it is held on and without a copy where the producer allows it; a NumPy array copied to the default device, as a learner reads it.

class source

class Branching(Protocol)

A simulator a cooldown branch steps and then leaves as the run left it: branch(), a context after which its environments stand as they did as it was entered. They live outside JAX, where no rollout can keep them, so a simulator keeps them itself (branched).

def branch() -> contextlib.AbstractContextManager[None]

class source

class DrivenSimulator(simulator: Simulator)

simulator driven as Driven drives an Env, for a world-model learner: its observation is the whole step the learner records, a dict of the observation, the reward that came with it, and whether it is the episode’s first, its last or terminal; an episode’s last observation is answered as a step of its own, and the step after it, whatever its action, restarts the episode (DreamerV3’s driver, embodied/core/driver.py). It restarts its episodes itself: a rollout over it passes reset=False.

def reset(seed: int) -> Observation
def step(action: jax.Array) -> TimeStep
def branch() -> Iterator[None]

A cooldown branch’s steps (branched), after which the simulator and the episodes it is to restart stand as the run left them.

attribute source

An array as a simulator hands it over: a JAX or NumPy array, or another framework’s (Shared).

class source

class Shared(Protocol)

An array a simulator hands over: anything exposing the DLPack protocol (a PyTorch tensor, a Warp or CuPy array), which JAX takes without a copy where both sides hold it on one device.

class source

class Simulator(Stepped, Protocol)

A batch of environments stepped from the host (Stepped) whose step restarts none: restart(mask) starts a new episode in each environment mask [count] selects and answers every environment’s observation, the selected ones’ their new episodes’ first.

def restart(mask: jax.Array) -> Observation

class source

class Stepped(Protocol)

A batch of count environments stepped from the host. reset(seed) starts every episode and answers each first observation; step(action) advances every environment by its row of a [count, action_size] action and answers a TimeStep. Arrays cross by DLPack (Shared, received).

def reset(seed: int) -> Observation
def step(action: jax.Array) -> TimeStep

function source

def branched(simulator: Stepped) -> Iterator[None]

simulator stepped in a cooldown branch and then left as the run left it, by its own branch() (Branching); a simulator with none is refused the branch.

function source

def received(tree: Handed | Tree) -> jax.Array | Tree

A simulator’s array, or a tree of them (an observation, a step, a state), each array taken as a JAX array (taken).

function source

def rollout_start(env: Env | Simulator, key: jax.Array, memory: Memory = None) -> Carry

A rollout’s carry at fresh episodes of env, an Env (start) or a Simulator (simulator_start).

function source

def simulator_rollout(
simulator: Stepped,
policy: Policy,
params: Params,
carry: Carry,
steps: int,
*,
reset: bool = True,
models: tuple[Held, ...] = (),
) -> tuple[Carry, Transition]

steps steps of policy under params in simulator from carry: the policy compiled as a program (dew.nn.kernels.options.program, once for each policy, under the options of the models it runs) and fetched once, so no step builds its dispatch key (OuterProgram.executable); the simulator stepped from the host, each step’s Transition stacked along a leading step axis [steps, ...], as env_rollout records them; the carry’s state is the simulator’s own, held by it, so the carry holds None there. With reset, an episode that ended restarts before the next step, its transition keeping the observation it ended at, which takes a Simulator; one that restarts its episodes itself (DrivenSimulator) passes False. The policy acts where the simulator’s arrays are, its params and the carry’s key and memory put there whole (a no-op where they already are), replicated over a mesh the observations are laid out on, the observations keeping their layout.

function source

def simulator_start(simulator: Stepped, key: jax.Array, memory: Memory = None) -> Carry

A rollout’s carry at fresh episodes of simulator: its first observations, seeded from key, and the policy’s memory as it starts.

function source

def taken(leaf: Handed | np.generic) -> jax.Array

A simulator’s array as a JAX array: a device array through DLPack, on the device it is held on and without a copy where the producer allows it; a NumPy array copied to the default device, as a learner reads it.