dew.rl.simulator
A simulator outside JAX as an environment: the boundary any GPU simulator plugs into.
A JAX-native simulator is an Env and runs inside env_rollout’s scan. A simulator written in another
framework (a CUDA, Warp, PyTorch or C++ engine) cannot step inside a jitted program, so it is stepped
from the host, a batch at a time. Its arrays cross the boundary by DLPack, without a copy where both
sides hold them on one device: an action is a JAX array, which a simulator takes with its own
from_dlpack (torch.from_dlpack, wp.from_dlpack, cupy.from_dlpack); an observation, reward or flag
is any array exposing __dlpack__, which jax.dlpack.from_dlpack takes.
A simulator steps every environment and restarts the episodes it is told to (Simulator.restart), so a
driver decides when an ended episode restarts: simulator_rollout restarts it at once, recording what
env_rollout records, the same Transition and Carry; DrivenSimulator restarts it a step later, as
Driven drives an Env for a world-model learner. rollout_start starts either kind of environment.
A cooldown branch steps a rollout’s environments and then leaves them as the run left them
(dew.training.Rollout’s branch). An Env’s are arrays the rollout keeps; a simulator keeps its own,
which it puts back where it has a branch() (Branching), and one with none is refused the branch.
| Name | Summary |
|---|---|
Branching | A simulator a cooldown branch steps and then leaves as the run left it: branch(), a context after which its environments stand as they did as it was entered. |
DrivenSimulator | simulator driven as Driven drives an Env, for a world-model learner: its observation is the whole step the learner records, a dict of the observation, the reward that came with it, and whether it is the episode’s first, its last or terminal; an episode’s last observation is answered as a step of its own, and the step after it, whatever its action, restarts the episode (DreamerV3’s driver, embodied/core/driver.py). |
Handed | An array as a simulator hands it over: a JAX or NumPy array, or another framework’s (Shared). |
Shared | An array a simulator hands over: anything exposing the DLPack protocol (a PyTorch tensor, a Warp or CuPy array), which JAX takes without a copy where both sides hold it on one device. |
Simulator | A batch of environments stepped from the host (Stepped) whose step restarts none: restart(mask) starts a new episode in each environment mask [count] selects and answers every environment’s observation, the selected ones’ their new episodes’ first. |
Stepped | A batch of count environments stepped from the host. |
branched | simulator stepped in a cooldown branch and then left as the run left it, by its own branch() (Branching); a simulator with none is refused the branch. |
received | A simulator’s array, or a tree of them (an observation, a step, a state), each array taken as a JAX array (taken). |
rollout_start | A rollout’s carry at fresh episodes of env, an Env (start) or a Simulator (simulator_start). |
simulator_rollout | steps steps of policy under params in simulator from carry: the policy compiled as a program (dew.nn.kernels.options.program, once for each policy, under the options of the models it runs) and fetched once, so no step builds its dispatch key (OuterProgram.executable); the simulator stepped from the host, each step’s Transition stacked along a leading step axis [steps, ...], as env_rollout records them; the carry’s state is the simulator’s own, held by it, so the carry holds None there. |
simulator_start | A rollout’s carry at fresh episodes of simulator: its first observations, seeded from key, and the policy’s memory as it starts. |
taken | A simulator’s array as a JAX array: a device array through DLPack, on the device it is held on and without a copy where the producer allows it; a NumPy array copied to the default device, as a learner reads it. |
Branching
Section titled “Branching”class Branching(Protocol)A simulator a cooldown branch steps and then leaves as the run left it: branch(), a context after
which its environments stand as they did as it was entered. They live outside JAX, where no rollout
can keep them, so a simulator keeps them itself (branched).
Branching.branch
Section titled “Branching.branch”def branch() -> contextlib.AbstractContextManager[None]DrivenSimulator
Section titled “DrivenSimulator”class DrivenSimulator(simulator: Simulator)simulator driven as Driven drives an Env, for a world-model learner: its observation is the
whole step the learner records, a dict of the observation, the reward that came with it, and
whether it is the episode’s first, its last or terminal; an episode’s last observation is
answered as a step of its own, and the step after it, whatever its action, restarts the episode
(DreamerV3’s driver, embodied/core/driver.py). It restarts its episodes itself: a rollout over it
passes reset=False.
DrivenSimulator.reset
Section titled “DrivenSimulator.reset”def reset(seed: int) -> ObservationDrivenSimulator.step
Section titled “DrivenSimulator.step”def step(action: jax.Array) -> TimeStepDrivenSimulator.branch
Section titled “DrivenSimulator.branch”def branch() -> Iterator[None]A cooldown branch’s steps (branched), after which the simulator and the episodes it is to
restart stand as the run left them.
Handed
Section titled “Handed”An array as a simulator hands it over: a JAX or NumPy array, or another framework’s (Shared).
Shared
Section titled “Shared”class Shared(Protocol)An array a simulator hands over: anything exposing the DLPack protocol (a PyTorch tensor, a Warp or CuPy array), which JAX takes without a copy where both sides hold it on one device.
Simulator
Section titled “Simulator”class Simulator(Stepped, Protocol)A batch of environments stepped from the host (Stepped) whose step restarts none:
restart(mask) starts a new episode in each environment mask [count] selects and answers every
environment’s observation, the selected ones’ their new episodes’ first.
Simulator.restart
Section titled “Simulator.restart”def restart(mask: jax.Array) -> ObservationStepped
Section titled “Stepped”class Stepped(Protocol)A batch of count environments stepped from the host. reset(seed) starts every episode and
answers each first observation; step(action) advances every environment by its row of a
[count, action_size] action and answers a TimeStep. Arrays cross by DLPack (Shared,
received).
Stepped.reset
Section titled “Stepped.reset”def reset(seed: int) -> ObservationStepped.step
Section titled “Stepped.step”def step(action: jax.Array) -> TimeStepbranched
Section titled “branched”def branched(simulator: Stepped) -> Iterator[None]simulator stepped in a cooldown branch and then left as the run left it, by its own branch()
(Branching); a simulator with none is refused the branch.
received
Section titled “received”def received(tree: Handed | Tree) -> jax.Array | TreeA simulator’s array, or a tree of them (an observation, a step, a state), each array taken as a JAX
array (taken).
rollout_start
Section titled “rollout_start”def rollout_start(env: Env | Simulator, key: jax.Array, memory: Memory = None) -> CarryA rollout’s carry at fresh episodes of env, an Env (start) or a Simulator
(simulator_start).
simulator_rollout
Section titled “simulator_rollout”def simulator_rollout( simulator: Stepped, policy: Policy, params: Params, carry: Carry, steps: int, *, reset: bool = True, models: tuple[Held, ...] = (),) -> tuple[Carry, Transition]steps steps of policy under params in simulator from carry: the policy compiled as a program
(dew.nn.kernels.options.program, once for each policy, under the options of the models it runs) and
fetched once, so no step builds its dispatch key (OuterProgram.executable); the simulator stepped
from the host, each step’s Transition stacked along a leading step axis [steps, ...], as
env_rollout records them; the carry’s state is the simulator’s own, held by it, so the carry holds
None there. With reset, an episode that ended restarts before the next step, its transition keeping
the observation it ended at, which takes a Simulator; one that restarts its episodes itself
(DrivenSimulator) passes False. The policy acts where the simulator’s arrays are, its params and
the carry’s key and memory put there whole (a no-op where they already are), replicated over a mesh
the observations are laid out on, the observations keeping their layout.
simulator_start
Section titled “simulator_start”def simulator_start(simulator: Stepped, key: jax.Array, memory: Memory = None) -> CarryA rollout’s carry at fresh episodes of simulator: its first observations, seeded from key,
and the policy’s memory as it starts.
def taken(leaf: Handed | np.generic) -> jax.ArrayA simulator’s array as a JAX array: a device array through DLPack, on the device it is held on and without a copy where the producer allows it; a NumPy array copied to the default device, as a learner reads it.