dew.nn.dex
Differential extension (DEX, arXiv 2505.16333): a learned differential on chosen heads’ outputs.
On each chosen head h of a softmax attention layer, the head’s output O_h (its attention-weighted
values, before the output projection) becomes
O'_h = O_h - lambda(t) * O_h W_D,h (paper Eq. 3, 5)with W_D,h a learned [Dv, Dv] projection of the head’s own, and the other heads unchanged. lambda is
annealed over the first anneal_steps optimizer updates t from zero to a learned scalar (Eq. 4):
lambda(t) = (1 - a) * (t / T) * lambda_init + a * lambda_learn, a = min(t / T, 1)so a pretrained model computes its own function at the start (lambda(0) = 0) and the extension enters
as it trains. lambda_init defaults to Differential Transformer’s depth schedule,
0.8 - 0.6 exp(-0.3 (l - 1)) for the layer’s 1-based depth l (arXiv 2410.05258, which DEX adopts).
Outside training (no step, dew.nn.progress) a layer reads lambda_learn, the anneal’s end.
The heads are chosen per layer (heads), as the paper picks the half of each layer’s heads of least
importance or most attention entropy; a run picks them from a model’s own statistics before it attaches
the extension. A layer holds its chosen heads and its lambda_init as variables of its own
(DEX_CONSTANTS, DexStart), so layers that choose as many heads share one spec (LayerDex) and a
scanned run or a pipeline stage holds them as it holds any layer’s weights. The paper’s recipe trains
the attention’s key, value and output projections with the extension and freezes the rest
(dew.optim.ParamGroup).
| Name | Summary |
|---|---|
DEX_COLLECTION | The collection an attention layer holds DEX’s start in (DEX_CONSTANTS), which never trains. |
DEX_CONSTANTS | The variables an attention layer holds for DEX under DEX_COLLECTION, by name: the heads it acts on and its lambda_init (DexStart). |
DEX_PARAMETERS | The parameters an attention layer adds for DEX, by name: its chosen heads’ projections and its learned lambda (dew.nn.mixers.attention.CausalSelfAttention). |
DexStart | The values one layer’s extension holds from its start (DEX_CONSTANTS): the heads it acts on, ascending, and its lambda_init. |
DifferentialExtension | Which heads of which layers carry the differential extension (this module’s doc), as (layer, heads) pairs, each layer once; lambda_init for every chosen layer, or None for Differential Transformer’s depth schedule (diff_lambda_init); and the optimizer updates lambda anneals over. |
HeadOutputs | A mixer whose heads’ outputs DEX extends, before its output projection: softmax attention (dew.nn.mixers.attention.CausalSelfAttention). |
LayerDex | One layer’s differential extension as every layer that chooses as many heads shares it: how many of its heads it acts on and the updates its lambda anneals over (DifferentialExtension). |
annealed | lambda(t) after updates optimizer updates (this module’s Eq. |
diff_lambda_init | Differential Transformer’s lambda_init for a layer of 1-based depth: 0.8 - 0.6 exp(-0.3 (l - 1)). |
differential | The heads’ outputs attention [B, S, H, Dv] with each of heads less strength times its own projection of itself (projection [len(heads), Dv, Dv], in heads’ order), the other heads as they are: one product over the chosen heads, scattered back. |
folded | variables with each DEX layer’s differential at its learned lambda folded into its output projection, the extension’s parameters and constants dropped: a layer’s o_proj kernel [H Dv, D] has each chosen head h’s rows K_h made (I - lambda_learn W_h) K_h, in float64 and rounded once to the kernel’s dtype, so the source’s own layer computes what the extended one does outside a run (annealed), O'_h K_h = O_h (I - lambda W_h) K_h. |
DEX_COLLECTION
Section titled “DEX_COLLECTION”DEX_COLLECTION = 'constants'The collection an attention layer holds DEX’s start in (DEX_CONSTANTS), which never trains.
DEX_CONSTANTS
Section titled “DEX_CONSTANTS”DEX_CONSTANTS = ('dex_heads', 'dex_lambda_init')The variables an attention layer holds for DEX under DEX_COLLECTION, by name: the heads it acts on and
its lambda_init (DexStart).
DEX_PARAMETERS
Section titled “DEX_PARAMETERS”DEX_PARAMETERS = ('dex_projection', 'dex_lambda')The parameters an attention layer adds for DEX, by name: its chosen heads’ projections and its learned
lambda (dew.nn.mixers.attention.CausalSelfAttention).
DexStart
Section titled “DexStart”class DexStart(heads: tuple[int, ...], lambda_init: float)The values one layer’s extension holds from its start (DEX_CONSTANTS): the heads it acts on,
ascending, and its lambda_init.
DifferentialExtension
Section titled “DifferentialExtension”class DifferentialExtension( layers: tuple[tuple[int, tuple[int, ...]], ...], lambda_init: float | None = None, anneal_steps: int = 1000, trains: Literal['attention', 'all'] = 'attention',)Which heads of which layers carry the differential extension (this module’s doc), as
(layer, heads) pairs, each layer once; lambda_init for every chosen layer, or None for
Differential Transformer’s depth schedule (diff_lambda_init); and the optimizer updates
lambda anneals over.
trains: Literal['attention', 'all']-
What of the model trains beside the extension: the paper’s recipe, every attention layer’s key, value and output projections, the rest frozen (
trainable); or all of it.
DifferentialExtension.trainable
Section titled “DifferentialExtension.trainable”def trainable( path: tuple[str, ...], decoder: tuple[str, ...], attention: Callable[[str], bool],) -> boolWhether the parameter at path (("params", ...)) trains under the extension’s recipe
(trains): every parameter with “all”; else the extension’s own, and the key, value and output
projections of the decoder’s softmax attention layers, whose modules’ names attention takes
(layers_3, or a scanned run’s layers_3_7), under the decoder’s parameters (decoder,
("params",) for a bundle’s own decoder, deeper in a wrapper’s).
DifferentialExtension.layer
Section titled “DifferentialExtension.layer”def layer(index: int) -> LayerDex | NoneLayer index’s extension as its spec holds it, or None where it carries none.
DifferentialExtension.start
Section titled “DifferentialExtension.start”def start(index: int) -> DexStart | NoneThe values layer index’s extension starts from, or None where it carries none.
HeadOutputs
Section titled “HeadOutputs”class HeadOutputs(Protocol)A mixer whose heads’ outputs DEX extends, before its output projection: softmax attention
(dew.nn.mixers.attention.CausalSelfAttention).
HeadOutputs.head_outputs
Section titled “HeadOutputs.head_outputs”def head_outputs() -> tuple[int, int]Its heads’ count and the width of each one’s output.
HeadOutputs.dex_fold_refusal
Section titled “HeadOutputs.dex_fold_refusal”def dex_fold_refusal() -> str | NoneWhy DEX on its heads does not fold into its output projection (folded): what its heads’
outputs pass through between the two; None where nothing does.
LayerDex
Section titled “LayerDex”class LayerDex(count: int, anneal_steps: int)One layer’s differential extension as every layer that chooses as many heads shares it: how many
of its heads it acts on and the updates its lambda anneals over (DifferentialExtension). Which
heads, and lambda’s start, are the layer’s own variables (DexStart).
annealed
Section titled “annealed”def annealed( updates: jax.Array | None, layer: LayerDex, start: jax.Array, learned: jax.Array,) -> jax.Arraylambda(t) after updates optimizer updates (this module’s Eq. 4), from the layer’s
lambda_init start; learned past the anneal, and outside training (updates None).
diff_lambda_init
Section titled “diff_lambda_init”def diff_lambda_init(depth: int) -> floatDifferential Transformer’s lambda_init for a layer of 1-based depth:
0.8 - 0.6 exp(-0.3 (l - 1)).
differential
Section titled “differential”def differential( attention: jax.Array, heads: jax.Array | tuple[int, ...], projection: jax.Array, strength: jax.Array, precision=None,) -> jax.ArrayThe heads’ outputs attention [B, S, H, Dv] with each of heads less strength times its own
projection of itself (projection [len(heads), Dv, Dv], in heads’ order), the other heads as
they are: one product over the chosen heads, scattered back.
folded
Section titled “folded”def folded( variables: Mapping[str, Mapping], compute: DTypeLike | None,) -> dict[str, Mapping]variables with each DEX layer’s differential at its learned lambda folded into its output
projection, the extension’s parameters and constants dropped: a layer’s o_proj kernel
[H Dv, D] has each chosen head h’s rows K_h made (I - lambda_learn W_h) K_h, in float64
and rounded once to the kernel’s dtype, so the source’s own layer computes what the extended one
does outside a run (annealed), O'_h K_h = O_h (I - lambda W_h) K_h. Lambda and W_h are the
ones the layer multiplies by, rounded to its compute dtype as differential rounds them (None for
its parameters’ own). A layer whose heads’ outputs pass through more before o_proj does not fold
(HeadOutputs.dex_fold_refusal). Each layer’s lambda, W_h and heads are read home as a pool’s
export reads them (read_home), the layers in order, the same on every process, and each kernel
folds as it is read (_fold, a Deferred leaf), so a host holds one folded kernel, not every
layer’s.