Skip to content

dew.rl.replay

A replay of environment steps held on the device, written and sampled by jittable functions.

The replay keeps the last capacity steps of each of envs environments, a ring per environment over a pytree of step fields [capacity, envs, ...], its leading axes shardable over the environment axis as the batch is. sample draws uniformly among the windows of consecutive steps the replay holds whole (DreamerV3’s uniform replay, arXiv 2301.04104, of length-step chunks).

NameSummary
ReplayBufferThe replay’s state: the held steps’ leaves [capacity, envs, ...], the slot the next step writes (cursor) and the steps written so far, at most capacity (filled).
StepsA pytree of step fields, each leaf leading with its step and environment axes.

dataclass source

class ReplayBuffer()

The replay’s state: the held steps’ leaves [capacity, envs, ...], the slot the next step writes (cursor) and the steps written so far, at most capacity (filled).

def empty(example: Steps, capacity: int) -> ReplayBuffer

An empty replay of capacity steps of what example, one step [envs, ...], holds.

def add(steps: Steps) -> ReplayBuffer

The replay after steps, leaves [T, envs, ...] in order, the oldest overwritten.

def sample(key: jax.Array, batch: int, length: int) -> Steps

batch windows of length consecutive steps [batch, length, ...], each from an environment and a start drawn uniformly among those the replay holds whole, its steps in the order they were written. The replay must hold length steps.

attribute source

A pytree of step fields, each leaf leading with its step and environment axes.