Skip to content

dew.nn.memory

In-context memory: a layer whose inner network’s parameters update over the sequence (ledger 1645).

A token’s views of the stream, k_t, v_t, q_t, train the inner network f(W; .) on an objective of its own, and the token reads the updated network:

g_t = grad_W objective(f(W_t'; k_t), target_t) gradients at the chunk's start state W_t'
S_t = mu_t S_{t-1} - theta_t g_t momentum (mu = 0 without it)
W_t = a_t W_{t-1} + S_t retention (a = 1 - alpha; 1 without it)
y_t = f(W_t; q_t) or f(W_t'; q_t) where a chunk commits at its end

(Titans, arXiv 2501.00663, Eq 8-15; TTT, arXiv 2407.04620, Eq 2-6; Miras, arXiv 2504.13173, Eq 10, 13, 28; Nested Learning, arXiv 2512.24695, Eq 71, 84-90.) The target is the value view (given) or the network’s own image of it at the chunk’s start, f(W_t'; v_t) (self_generated, Nested Learning Eq 84). A chunk of chunk tokens takes every gradient at the state it starts from, counted from its document’s first real token; chunk=1 is online gradient descent. Each token reads the network its own step updated, or, where the chunk commits its steps at its end (commit), the network its chunk started from (a continuum-memory level, Nested Learning Eq 71). A document’s first token starts from W_0, the network’s meta-learned parameters, with no momentum; the parameters the inner loop never steps (slow) stay at W_0 throughout.

The state (W, S and the chunk’s start) is held in at least float32 whatever the compute dtype, and the inner gradients and the scans run there: a bfloat16 state drops every step below its ulp. The readout runs the network at the layer’s dtype over the state cast to it, so at a zero step size a network reads exactly as its W_0 does.

Two computations give the one function. A row of one document runs chunk by chunk: its real tokens compacted to the front (valid_order), each chunk’s gradients one vmap at the chunk’s start, the momentum and retention across the chunk one associative scan of their affine steps over every leaf at once (the leaves laid side by side, _flattened), and the chunk’s readouts one vmap over its states. A packed row, and a cached call, run token by token (_stepped), which is the definition. The state is the network’s parameters, a traced carry. Each head’s network is this layer’s child (head_0, …), whose parameters are its W_0, and every application runs that child at its own scope over the weights the step holds, read-only (_substituted), so the layer composes with a scanned layer stack and remat, and a method interceptor the model computes under finds each head’s sites at their paths, as at any child; the chunk’s carry is named (CARRY) for a remat policy. The heads run one after another, each vectorized over its rows and tokens, and may hold different trees (an adapter on one head alone).

An adapter on an inner kernel (dew.lora) adds its branch, s drop(x) A B, at every application, as it does at any Dense, its dropout drawing from each token’s key in a forward that carries the dropout stream. Its update is W_0’s: the B factor rides the state with no step of its own, so a retention forgets it as it forgets W_0, and the memory computes what the adapter’s merge computes. A checkpoint’s input quantization at an inner site (dew.quantization) rounds that site’s input at every application alike, and Qwix’s quantized training quantizes every product of the moving weights, its stochastic rounding drawing from each token’s key. A frozen FP8 record’s codes at an inner kernel stand for W_0’s kernel, as they decode: in a memory only W_0 is frozen, and each application multiplies the state the steps move from it. Qwix’s serving quantization stores W_0 quantized, which the steps start from as the float it decodes to (dew.quantization.decoded_stored); each product of the moving weights rounds as quantized training rounds it (dew.quantization.moving_weights), so a served memory computes what the quantized memory computes from the stored W_0.

A linear memory (LinearMemory) on its given target, with no momentum, no orthogonalized or slow weights and each token reading its own step, whose heads each compute the linear map of their input by the kernel its state moves (no wrapper rounds or drops their inputs or quantizes their products, _route), is a row of the shared linear recurrence (dew.nn.kernels.recurrence), and runs there: on the dot objective its step writes theta k v^T, its retention the decay, at any chunk (the gradient is the same at every state); on the l2 objective at chunk 1 without retention it is the delta rule with beta theta (Titans takes the gradient before the retention, the gated delta rule after it, so those differ). A call runs the recurrence’s chunked op (recurrence_chunk.chunk, on its kernels where they take the call) and a cached token its decode step (recurrence.step). The recurrence starts each document from zero, so W_0 enters as its own term: the delta rule’s state is W - W_0, written with the values v - W_0^T k, and the dot rule’s W - P_t W_0, P_t the product of the shares the document kept; each readout adds P_t W_0^T q (P_t one for the delta rule). That readout is computed in at least float32 and rounded once to the layer’s dtype. Its decode cache holds the recurrence’s state whichever route a call takes, as a call’s wrappers may take it off the recurrence (an adapter’s dropout on a training forward), whose general path steps the weights that state makes (_stepped_from_recurrence).

NameSummary
CARRYThe name of the state a chunk hands the next (checkpoint_name), for a remat policy to save (dew.nn.mixer_base.RESIDUALS).
GateA per-token scalar a head: value itself, or, without one, scale * sigmoid(x w + b) of the token’s stream (TTT’s eta(x) = eta_base sigmoid(theta_lr x), Sec 2.7), b starting at bias.
InContextMemoryThe in-context memory layer [B, S, D] -> [B, S, D] (the module docstring).
LinearMemoryW k, W square over the view’s width: a linear memory (TTT-Linear’s f_lin without its norm and residual, Titans’ linear case).
MLPMemoryTitans’ deep memory: dense layers of the hidden widths, then one back to the view’s width, activation between them (Sec 3.1), layers_{i} each: each two layers in turn an ungated GatedMLP, a last odd one a Dense.
ObjectiveThe loss a token’s prediction f(W; k) takes against its target, summed over the features: dot, -<p, v> (linear attention’s, Nested Learning Eq 17); l2, 1/2 ||p - v||^2 (Titans Eq 12 with Appendix C’s 1/2); lp, ||p - v||_p^p (Miras Eq 10); huber, Huber’s loss of each feature’s error at delta (Miras Eq 13); l2_or_l1, the l2 loss while the whole error’s norm is at most delta, else delta times its l1 norm (Miras Eq 16, Yaad’s Eq 26), so past delta every feature steps delta times its error’s sign, however small its own error.
ResidualMemoryx + N(f_res(x)): f_res linear (hidden None) or `Dense(D) .

attribute source

CARRY = MEMORY_RESIDUALS[0]

The name of the state a chunk hands the next (checkpoint_name), for a remat policy to save (dew.nn.mixer_base.RESIDUALS).

dataclass source

class Gate(value: float | None = None, scale: float = 1.0, bias: float = 0.0)

A per-token scalar a head: value itself, or, without one, scale * sigmoid(x w + b) of the token’s stream (TTT’s eta(x) = eta_base sigmoid(theta_lr x), Sec 2.7), b starting at bias.

class source

class InContextMemory(LinearProducts)

The in-context memory layer [B, S, D] -> [B, S, D] (the module docstring).

network is the inner network, any module that maps one vector of the view’s width to one of the same width and takes a dtype (LinearMemory, ResidualMemory, MLPMemory, an existing block such as GatedMLP); it runs as each head’s scope stores it, at the dtype each use asks, and its parameters, drawn at init, are W_0. heads memories run side by side over the views split into heads of head_features. A view is projected from the stream, or the stream itself where its project_* is off, as a continuum-memory level reads its block’s input (one head). gates are the step size theta (learning_rate), the momentum mu (None: none) and the retention alpha (None: none). output projects the heads’ readouts back to the stream’s width.

commit: Literal['token', 'chunk']

When a step becomes the network a token reads: each token’s at once (token, TTT and Titans, which read the state their own step updated), or a chunk’s at its end (chunk): the network stays as it is between those scheduled updates, so every token of a chunk reads the network its chunk started from, a continuum-memory level’s (Nested Learning Eq 71).

orthogonalize: int

Newton-Schulz steps orthogonalizing each matrix of a token’s gradient before its step, Muon’s own iteration (dew.newton_schulz.orthogonalized_leaf, Nested Learning Sec 4.2’s Muon-style inner update); 0 steps the gradient as it is. Vectors (biases, norm scales) step as they are.

slow: tuple[str, ...]

The network’s parameters, by their top-level names (a submodule’s or a parameter’s), that the inner loop reads at W_0 and never steps: meta-learned by the outer loop alone, as TTT’s layer norm is (the authors’ ttt.py updates only ttt_param_names, W1 and b1, and W2 and b2). The rest are the fast weights the sequence updates.

residuals: tuple[str, ...]

The residuals this layer may mark for a remat policy (dew.nn.mixer_base.RESIDUALS): those of every path its configuration can take, as the call’s wrappers decide between them and the configuration does not see the wrappers (_route). A linear memory on the shared recurrence marks its rule’s (the delta rule’s port saves its decay, inverse and memory; the dot rule’s portable math marks none), and on the general path, as any other memory, its chunk scan’s carry.

def setup()

Flax module source

class LinearMemory(dtype: Dtype | None = None, precision: PrecisionLike = None)

W k, W square over the view’s width: a linear memory (TTT-Linear’s f_lin without its norm and residual, Titans’ linear case).

Flax module source

class MLPMemory(
activation: Literal['gelu', 'silu', 'relu'] = 'silu',
dtype: Dtype | None = None,
precision: PrecisionLike = None,
)

Titans’ deep memory: dense layers of the hidden widths, then one back to the view’s width, activation between them (Sec 3.1), layers_{i} each: each two layers in turn an ungated GatedMLP, a last odd one a Dense. activation names one of dew.nn.activations.UNGATED.

dataclass source

class Objective(
kind: Literal['dot', 'l2', 'lp', 'huber', 'l2_or_l1'] = 'l2',
p: float = 3.0,
delta: float = 1.0,
)

The loss a token’s prediction f(W; k) takes against its target, summed over the features: dot, -<p, v> (linear attention’s, Nested Learning Eq 17); l2, 1/2 ||p - v||^2 (Titans Eq 12 with Appendix C’s 1/2); lp, ||p - v||_p^p (Miras Eq 10); huber, Huber’s loss of each feature’s error at delta (Miras Eq 13); l2_or_l1, the l2 loss while the whole error’s norm is at most delta, else delta times its l1 norm (Miras Eq 16, Yaad’s Eq 26), so past delta every feature steps delta times its error’s sign, however small its own error.

Flax module source

class ResidualMemory(
hidden: int | None = None,
activation: Literal['gelu', 'silu', 'relu'] = 'gelu',
norm: bool = True,
dtype: Dtype | None = None,
precision: PrecisionLike = None,
)

x + N(f_res(x)): f_res linear (hidden None) or Dense(D) . activation . Dense(hidden), the ungated GatedMLP (up, down), N a layer norm or none. TTT’s f (Sec 2.7: LN and residual, f_lin or a 4x GELU MLP), Nested Learning’s memories (Eq 89, no norm) and Miras’ (Sec 5.3). activation names one of dew.nn.activations.UNGATED.