Skip to content

dew.objectives.rl.tssm

TransDreamer’s transformer state-space model (Chen et al., arXiv 2202.09481): a Dreamer world model’s dynamics as a causal transformer over the latents and actions before each step.

At step t the posterior reads the observation’s tokens alone, q(z_t | x_t), so a window’s latents are drawn at once; the transformer reads each step’s previous latent and action, (z_{t-1}, a_{t-1}), and answers the deterministic state h_t, from which the prior predicts z_t, p(z_t | h_t). h_t therefore holds nothing of x_t, and the prior is a prediction. Attention stays within an episode: a step that starts one reads a learned start embedding and attends to nothing before it. Training runs the window in parallel; acting and imagination run the same attention a step at a time over a cache of keys and values (Cache), so the policy and the imagined rollouts read the function the model was trained as.

The sizes default to FlyDreamer’s (deter 512, 32 categoricals of 32 classes, 4 layers of 8 heads, rotary positions, RMS-normed queries and keys, a gated feed-forward four widths wide before its 2/3).

NameSummary
TSSMTransDreamer’s dynamics (module docstring) over tokens [..., E] and actions of action_size.
CacheThe keys and values a step attends to, [B, layers, C, heads, head] each, with whether each slot is valid [B, C], the slot the next step writes (index, wrapping round the C slots) and that step’s position within its episode [B]; for imagination, beside them the window its starts share.
WindowA window’s keys and values [B, layers, T, heads, head] that imagination from each of its B * T steps attends to, held once for all of them, and which of the window’s steps each start reads [B * T, T] (those of its episode up to itself).

Flax module source

class TSSM(
stoch: int = 32,
classes: int = 32,
width: int = 512,
layers: int = 4,
heads: int = 8,
ffup: int = 4,
unimix: float = 0.01,
context: int = 64,
dtype: Dtype = jnp.float32,
)

TransDreamer’s dynamics (module docstring) over tokens [..., E] and actions of action_size.

context: int

The steps an acting cache holds (empty_cache), the windows’ length it trained on.

dtype: Dtype

The computation’s dtype, bfloat16 for speed; parameters stay at least float32 (at_least_fp32), so a bfloat16 computation updates float32 weights (DreamerV3’s COMPUTE_DTYPE beside its float32 parameters), and the latents’ distributions are at least float32.

def setup() -> None
def posterior(tokens: jax.Array) -> jax.Array

q(z_t | x_t): the latent’s log-probabilities from an observation’s tokens alone.

def prior(deter: jax.Array) -> jax.Array

p(z_t | h_t): the latent’s log-probabilities predicted from the state.

def inputs(stoch: jax.Array, previous: jax.Array, first: jax.Array) -> jax.Array

Each step’s input [..., width]: its previous latent stoch [..., S, K] (each step’s own, shifted by one along the step axis by the caller) and action, embedded, or the start embedding where the step first starts an episode.

def states(x: jax.Array, first: jax.Array) -> jax.Array

The states h [B, T, width] of a window’s inputs x (window’s).

def window(x: jax.Array, first: jax.Array) -> tuple[jax.Array, Window]

The states h [B, T, width] of a window’s inputs x [B, T, width], each step attending to the steps of its episode up to itself, the window’s first step starting its own segment; and the window’s keys and values, which imagination from each of its steps attends to (Window, window_cache).

def window_cache(window: Window, first: jax.Array, horizon: int) -> Cache

For each step k of a window [B, T], the cache imagination from it attends to: the window’s keys and values (window’s, held once), of which k reads its episode’s steps up to k, and horizon slots of its own for the steps it imagines, the next at the position after k’s.

def empty_cache(rows: int, size: int) -> Cache

A cache of size slots holding nothing, for acting from fresh episodes (context slots). Within an episode’s first size steps acting reads what training’s window does; past them the cache holds the last size steps’ keys, each computed over the context before it.

def step(x: jax.Array, first: jax.Array, cache: Cache) -> tuple[jax.Array, Cache]

One step’s state [B, width] from its input x [B, width], attending to the cache and to itself; a step that starts an episode first attends to nothing cached. The step’s keys and values are written at the cache’s next slot, the oldest where it is full.

dataclass source

class Cache(window: Window | None = None)

The keys and values a step attends to, [B, layers, C, heads, head] each, with whether each slot is valid [B, C], the slot the next step writes (index, wrapping round the C slots) and that step’s position within its episode [B]; for imagination, beside them the window its starts share.

dataclass source

class Window()

A window’s keys and values [B, layers, T, heads, head] that imagination from each of its B * T steps attends to, held once for all of them, and which of the window’s steps each start reads [B * T, T] (those of its episode up to itself).