dew.nn.memory
In-context memory: a layer whose inner network’s parameters update over the sequence (ledger 1645).
A token’s views of the stream, k_t, v_t, q_t, train the inner network f(W; .) on an objective
of its own, and the token reads the updated network:
g_t = grad_W objective(f(W_t'; k_t), target_t) gradients at the chunk's start state W_t'S_t = mu_t S_{t-1} - theta_t g_t momentum (mu = 0 without it)W_t = a_t W_{t-1} + S_t retention (a = 1 - alpha; 1 without it)y_t = f(W_t; q_t) or f(W_t'; q_t) where a chunk commits at its end(Titans, arXiv 2501.00663, Eq 8-15; TTT, arXiv 2407.04620, Eq 2-6; Miras, arXiv 2504.13173, Eq 10, 13,
28; Nested Learning, arXiv 2512.24695, Eq 71, 84-90.) The target is the value view (given) or the
network’s own image of it at the chunk’s start, f(W_t'; v_t) (self_generated, Nested Learning
Eq 84). A chunk of chunk tokens takes every gradient at the state it starts from, counted from its
document’s first real token; chunk=1 is online gradient descent. Each token reads the network its
own step updated, or, where the chunk commits its steps at its end (commit), the network its chunk
started from (a continuum-memory level, Nested Learning Eq 71). A document’s first token starts from
W_0, the network’s meta-learned parameters, with no momentum; the parameters the inner loop never
steps (slow) stay at W_0 throughout.
The state (W, S and the chunk’s start) is held in at least float32 whatever the compute dtype, and the inner gradients and the scans run there: a bfloat16 state drops every step below its ulp. The readout runs the network at the layer’s dtype over the state cast to it, so at a zero step size a network reads exactly as its W_0 does.
Two computations give the one function. A row of one document runs chunk by chunk: its real tokens
compacted to the front (valid_order), each chunk’s gradients one vmap at the chunk’s start, the
momentum and retention across the chunk one associative scan of their affine steps over every leaf at
once (the leaves laid side by side, _flattened), and the chunk’s readouts one vmap over its states.
A packed row, and a cached call, run token by token (_stepped), which is the definition. The state is
the network’s parameters, a traced carry. Each head’s network is this layer’s child (head_0, …),
whose parameters are its W_0, and every application runs that child at its own scope over the weights
the step holds, read-only (_substituted), so the layer composes with a scanned layer stack and remat,
and a method interceptor the model computes under finds each head’s sites at their paths, as at any
child; the chunk’s carry is named (CARRY) for a remat policy. The heads run one after another, each
vectorized over its rows and tokens, and may hold different trees (an adapter on one head alone).
An adapter on an inner kernel (dew.lora) adds its branch, s drop(x) A B, at every application, as it
does at any Dense, its dropout drawing from each token’s key in a forward that carries the dropout
stream. Its update is W_0’s: the B factor rides the state with no step of its own, so a retention
forgets it as it forgets W_0, and the memory computes what the adapter’s merge computes. A checkpoint’s
input quantization at an inner site (dew.quantization) rounds that site’s input at every application
alike, and Qwix’s quantized training quantizes every product of the moving weights, its stochastic
rounding drawing from each token’s key. A frozen FP8 record’s codes at an inner kernel stand for W_0’s
kernel, as they decode: in a memory only W_0 is frozen, and each application multiplies the state the
steps move from it. Qwix’s serving quantization stores W_0 quantized, which the steps start from as the
float it decodes to (dew.quantization.decoded_stored); each product of the moving weights rounds as
quantized training rounds it (dew.quantization.moving_weights), so a served memory computes what the
quantized memory computes from the stored W_0.
A linear memory (LinearMemory) on its given target, with no momentum, no orthogonalized or slow
weights and each token reading its own step, whose heads each compute the linear map of their input by
the kernel its state moves (no wrapper rounds or drops their inputs or quantizes their products,
_route), is a row of the shared linear recurrence (dew.nn.kernels.recurrence), and runs there: on the dot
objective its step writes theta k v^T, its retention the decay, at any chunk (the gradient is the same
at every state); on the l2 objective at chunk 1 without retention it is the delta rule with beta theta
(Titans takes the gradient before the retention, the gated delta rule after it, so those differ). A
call runs the recurrence’s chunked op (recurrence_chunk.chunk, on its kernels where they take the
call) and a cached token its decode step (recurrence.step). The recurrence starts each document from
zero, so W_0 enters as its own term: the delta rule’s state is W - W_0, written with the values
v - W_0^T k, and the dot rule’s W - P_t W_0, P_t the product of the shares the document kept; each
readout adds P_t W_0^T q (P_t one for the delta rule). That readout is computed in at least float32 and
rounded once to the layer’s dtype. Its decode cache holds the recurrence’s state whichever route a call
takes, as a call’s wrappers may take it off the recurrence (an adapter’s dropout on a training
forward), whose general path steps the weights that state makes (_stepped_from_recurrence).
| Name | Summary |
|---|---|
CARRY | The name of the state a chunk hands the next (checkpoint_name), for a remat policy to save (dew.nn.mixer_base.RESIDUALS). |
Gate | A per-token scalar a head: value itself, or, without one, scale * sigmoid(x w + b) of the token’s stream (TTT’s eta(x) = eta_base sigmoid(theta_lr x), Sec 2.7), b starting at bias. |
InContextMemory | The in-context memory layer [B, S, D] -> [B, S, D] (the module docstring). |
LinearMemory | W k, W square over the view’s width: a linear memory (TTT-Linear’s f_lin without its norm and residual, Titans’ linear case). |
MLPMemory | Titans’ deep memory: dense layers of the hidden widths, then one back to the view’s width, activation between them (Sec 3.1), layers_{i} each: each two layers in turn an ungated GatedMLP, a last odd one a Dense. |
Objective | The loss a token’s prediction f(W; k) takes against its target, summed over the features: dot, -<p, v> (linear attention’s, Nested Learning Eq 17); l2, 1/2 ||p - v||^2 (Titans Eq 12 with Appendix C’s 1/2); lp, ||p - v||_p^p (Miras Eq 10); huber, Huber’s loss of each feature’s error at delta (Miras Eq 13); l2_or_l1, the l2 loss while the whole error’s norm is at most delta, else delta times its l1 norm (Miras Eq 16, Yaad’s Eq 26), so past delta every feature steps delta times its error’s sign, however small its own error. |
ResidualMemory | x + N(f_res(x)): f_res linear (hidden None) or `Dense(D) . |
CARRY = MEMORY_RESIDUALS[0]The name of the state a chunk hands the next (checkpoint_name), for a remat policy to save
(dew.nn.mixer_base.RESIDUALS).
class Gate(value: float | None = None, scale: float = 1.0, bias: float = 0.0)A per-token scalar a head: value itself, or, without one, scale * sigmoid(x w + b) of the
token’s stream (TTT’s eta(x) = eta_base sigmoid(theta_lr x), Sec 2.7), b starting at bias.
InContextMemory
Section titled “InContextMemory”class InContextMemory(LinearProducts)The in-context memory layer [B, S, D] -> [B, S, D] (the module docstring).
network is the inner network, any module that maps one vector of the view’s width to one of the
same width and takes a dtype (LinearMemory, ResidualMemory, MLPMemory, an existing block
such as GatedMLP); it runs as each head’s scope stores it, at the dtype each use asks, and its
parameters, drawn at init, are W_0. heads memories run side by side over the views split into
heads of head_features. A view is projected from the stream, or the stream itself where its
project_* is off, as a continuum-memory level reads its block’s input (one head). gates are the
step size theta (learning_rate), the momentum mu (None: none) and the retention alpha (None: none).
output projects the heads’ readouts back to the stream’s width.
commit: Literal['token', 'chunk']-
When a step becomes the network a token reads: each token’s at once (
token, TTT and Titans, which read the state their own step updated), or a chunk’s at its end (chunk): the network stays as it is between those scheduled updates, so every token of a chunk reads the network its chunk started from, a continuum-memory level’s (Nested Learning Eq 71). orthogonalize: int-
Newton-Schulz steps orthogonalizing each matrix of a token’s gradient before its step, Muon’s own iteration (
dew.newton_schulz.orthogonalized_leaf, Nested Learning Sec 4.2’s Muon-style inner update); 0 steps the gradient as it is. Vectors (biases, norm scales) step as they are. slow: tuple[str, ...]-
The network’s parameters, by their top-level names (a submodule’s or a parameter’s), that the inner loop reads at W_0 and never steps: meta-learned by the outer loop alone, as TTT’s layer norm is (the authors’ ttt.py updates only
ttt_param_names, W1 and b1, and W2 and b2). The rest are the fast weights the sequence updates. residuals: tuple[str, ...]-
The residuals this layer may mark for a remat policy (
dew.nn.mixer_base.RESIDUALS): those of every path its configuration can take, as the call’s wrappers decide between them and the configuration does not see the wrappers (_route). A linear memory on the shared recurrence marks its rule’s (the delta rule’s port saves its decay, inverse and memory; the dot rule’s portable math marks none), and on the general path, as any other memory, its chunk scan’s carry.
InContextMemory.setup
Section titled “InContextMemory.setup”def setup()LinearMemory
Section titled “LinearMemory”class LinearMemory(dtype: Dtype | None = None, precision: PrecisionLike = None)W k, W square over the view’s width: a linear memory (TTT-Linear’s f_lin without its norm and
residual, Titans’ linear case).
MLPMemory
Section titled “MLPMemory”class MLPMemory( activation: Literal['gelu', 'silu', 'relu'] = 'silu', dtype: Dtype | None = None, precision: PrecisionLike = None,)Titans’ deep memory: dense layers of the hidden widths, then one back to the view’s width,
activation between them (Sec 3.1), layers_{i} each: each two layers in turn an ungated
GatedMLP, a last odd one a Dense. activation names one of dew.nn.activations.UNGATED.
Objective
Section titled “Objective”class Objective( kind: Literal['dot', 'l2', 'lp', 'huber', 'l2_or_l1'] = 'l2', p: float = 3.0, delta: float = 1.0,)The loss a token’s prediction f(W; k) takes against its target, summed over the features:
dot, -<p, v> (linear attention’s, Nested Learning Eq 17); l2, 1/2 ||p - v||^2 (Titans Eq 12
with Appendix C’s 1/2); lp, ||p - v||_p^p (Miras Eq 10); huber, Huber’s loss of each feature’s
error at delta (Miras Eq 13); l2_or_l1, the l2 loss while the whole error’s norm is at most
delta, else delta times its l1 norm (Miras Eq 16, Yaad’s Eq 26), so past delta every feature
steps delta times its error’s sign, however small its own error.
ResidualMemory
Section titled “ResidualMemory”class ResidualMemory( hidden: int | None = None, activation: Literal['gelu', 'silu', 'relu'] = 'gelu', norm: bool = True, dtype: Dtype | None = None, precision: PrecisionLike = None,)x + N(f_res(x)): f_res linear (hidden None) or Dense(D) . activation . Dense(hidden), the
ungated GatedMLP (up, down), N a layer norm or none. TTT’s f (Sec 2.7: LN and residual, f_lin
or a 4x GELU MLP), Nested Learning’s memories (Eq 89, no norm) and Miras’ (Sec 5.3). activation
names one of dew.nn.activations.UNGATED.