dew.nn.cross_attention
Gated cross-attention from a conditioning sequence into a decoder’s layers.
Flamingo’s block (Alayrac et al. 2022, section 2.2) runs before each decoder layer it names. The layer’s residual first reads the conditioning sequence through attention, then runs a feed-forward, and each output joins the residual through a tanh gate:
x = x + tanh(alpha_cross_attn) * cross_attn(input_layernorm(x), context)x = x + tanh(alpha_dense) * mlp(post_attention_layernorm(x))Both gates start at zero, so a pretrained decoder with the block attached
computes its own function until the gates train. The pretrained weights stay
frozen while the block trains (dew.interop.Pretrained.with_cross_attention).
The reference layer is IDEFICS’s IdeficsGatedCrossAttentionLayer
(transformers’ modeling_idefics.py), and its names are the names used here.
Its attention is the one every cross-attention runs, a diffusion block’s and
Wan’s alike (dew.nn.attention_blocks.NormalAttention), held in the block’s own
scope, over the decoder’s grouped key-value heads and keeping a cached call’s
keys and values; the gates, the feed-forward and the residual join are the
block’s.
A decoder call carries the sequence in its conditioning
(dew.nn.inputs.CONTEXT), with each row’s count of real tokens
(CONTEXT_LENGTHS). A row with no tokens adds nothing through either
branch, so it computes the decoder’s own function. IDEFICS still runs the
feed-forward for a token that attends to no image; here the block is absent
for such a row, as it is for a call with no context at all. A cached call
that carries the context keeps its keys and values in the cache, so later
decode steps attend over them without it.
That is one form of a decoder’s cross-attention (CrossBlocks, which a decoder’s
with_cross_attention takes): CrossAttention names it. SharedCrossAttention
is another, which adds no attention of its own: each named layer reads the
conditioning sequence through its own attention’s projections and norms,
unrotated and uncausal, and joins the read through one tanh gate, zero at the
start, the only weight it adds:
x = x + tanh(alpha_cross_attn) * self_attn.read(input_layernorm(x), context)Any configuration that builds a conforming block per layer is a third.
A block reads and joins whatever residual its decoder carries: the plain stream (AltUp’s active copy among them), mHC’s streams, whose mean it reads and each of which its outputs join, Kimi K3’s depth state, whose mixture it reads and whose partial block its outputs join, and nGPT’s unit stream, which it reads unnormed and which each output joins as a step of its gate along the sphere, as nGPT’s own sublayers do.
A gate multiplies its branch as an outer product, so the gate’s gradient, the sum of the branch times its cotangent over every token, is a dot: XLA:CPU reduces a broadcast’s transpose four times further from float64 than torch’s cascade sum over 640 terms, and its dot as close as torch’s.
| Name | Summary |
|---|---|
CONTEXT_LOGITS | The qk collection’s entry of each head’s largest logit over a context a layer’s own attention read (AttendsContext.attend_context), which the layer’s own maxima take in (with_context_logits). |
CROSS_CACHE | |
CROSS_LENGTHS | The cache variable of each row’s real context tokens, [rows]. |
NAME | The module name of a decoder layer’s cross-attention block, in every collection. |
AttendsContext | A layer’s own attention as a read of another sequence (SharedCrossAttention): whether it projects keys and values of its own to read one with, the key heads’ count and the widths of a key and a value head, its query heads over the layer’s input, its key and value heads over a context, its attention of the one over the other through its output projection, and that read’s work: dew.nn.mixers.attention.CausalSelfAttention’s, dew.nn.mla.MultiHeadLatentAttention’s (and so k-pool’s) and dew.nn.deepseek_v4.DeepseekV4Attention’s. |
ContextEncoder | How a context of token ids becomes the states a model’s cross-attention reads (encode_context): table, its tokens through the model’s own token table, or backbone, the model’s own final states over it in a pass of its own. |
CrossAttention | The decoder layers that read a conditioning sequence, and its geometry. |
CrossBlocks | A decoder’s cross-attention, which with_cross_attention takes: the layers that read a conditioning sequence, the most tokens a row’s context holds, which a decode cache keeps the keys and values of, the width of its tokens (None for the decoder’s own), and each layer’s block, built from the decoder. |
GatedCrossAttention | One layer’s gated block: a residual with its gated attention over a CrossContext and its gated feed-forward joined in. |
LayerReads | What a block that reads through its layer (reads_layer) is handed: the layer’s attention input from the states a sublayer reads, normed as its attention reads it, its attention, and the call’s progress (AttentionMetadata.progress), which that attention may read. |
SharedCrossAttention | Cross-attention through each named layer’s own attention: before its sublayers, the layer in layer_ids reads the conditioning sequence with its own query, key, value and output projections and norms, its logit scale, sinks and softcap, unrotated and uncausal (AttendsContext), and the read joins the residual through one tanh gate the layer adds, zero at the start, so the decoder computes its own function until the gates train. |
SharedGatedCrossAttention | One layer’s read of a conditioning sequence through its own attention (SharedCrossAttention): the residual’s states normed as the layer’s attention reads them, its query heads over them and its key and value heads over the context (AttendsContext), attended and projected by the layer’s own weights, the read joined through one tanh gate, alpha_cross_attn, zero at the start. |
context_mask | The keys a read of a context sees (AttendsContext.attend_context): query [B, S, H, D] over a context’s key [B, C, ..., D], each row’s first lengths keys real and, where given, each query its own document’s keys alone (queries [B, S], segments [B, C]), unmasked by causality. |
crossed_flops | decoder’s cross-attention blocks’ work over rows of lengths tokens reading contexts of context_lengths tokens (CrossBlocks.flops), or with held the part its own matrices do (CrossBlocks.held_flops), each block on every pass of its layer (layer_passes): none without cross-attention, and a call without a context runs none of it. |
crossing | Whether a variables path runs through a cross-attention block: the filter that trains the blocks alone (dew.batch.PathFilter). |
encode_context | The conditioning a model’s cross-attention layers read (CrossAttending) for contexts of token ids: tokens [B, C], each row’s first lengths real, as encoder says, and where a row packs the contexts of up to max_segments documents, each token’s document, segments [B, C] (CONTEXT_SEGMENTS), which the backbone pass reads each of alone and the call’s own tokens of each document read alone. |
gated_join | stream with branch joined through gate, in at least float32 of the three and rounded once to the stream’s dtype: added, broadcast over any axis the stream has beside the states’, or with ngpt moved by the step of gate from the unit stream toward the unit branch, nGPT’s step on the unit sphere, at the stream’s own length (_moved_on_the_sphere). |
sow_context_logits | A read of a context’s per-head largest logit over the keys it sees (context_mask), sown in module’s qk collection (CONTEXT_LOGITS) where a caller opened it, so the layer’s own call takes it in beside its own sequence’s (with_context_logits) and the QK-Clip bounds both. |
with_context_logits | found, a layer’s own call’s per-head largest logits, beside those of a context the call read first through the same projections (sow_context_logits). |
CONTEXT_LOGITS
Section titled “CONTEXT_LOGITS”CONTEXT_LOGITS = 'context_max_logits'The qk collection’s entry of each head’s largest logit over a context a layer’s own attention read
(AttendsContext.attend_context), which the layer’s own maxima take in (with_context_logits).
CROSS_CACHE
Section titled “CROSS_CACHE”CROSS_CACHE = frozenset({CROSS_KEY, CROSS_VALUE, CROSS_LENGTHS})CROSS_LENGTHS
Section titled “CROSS_LENGTHS”CROSS_LENGTHS = 'cross_lengths'The cache variable of each row’s real context tokens, [rows].
NAME = 'cross_attention'The module name of a decoder layer’s cross-attention block, in every collection.
AttendsContext
Section titled “AttendsContext”class AttendsContext(Protocol)A layer’s own attention as a read of another sequence (SharedCrossAttention): whether it
projects keys and values of its own to read one with, the key heads’ count and the widths of a
key and a value head, its query heads over the layer’s input, its key and value heads over a
context, its attention of the one over the other through its output projection, and that read’s
work: dew.nn.mixers.attention.CausalSelfAttention’s, dew.nn.mla.MultiHeadLatentAttention’s (and
so k-pool’s) and dew.nn.deepseek_v4.DeepseekV4Attention’s.
AttendsContext.context_heads
Section titled “AttendsContext.context_heads”def context_heads() -> tuple[int, int, int]AttendsContext.context_flops
Section titled “AttendsContext.context_flops”def context_flops(lengths, context_lengths) -> floatAttendsContext.context_projection_flops
Section titled “AttendsContext.context_projection_flops”def context_projection_flops(lengths, context_lengths) -> floatAttendsContext.context_query
Section titled “AttendsContext.context_query”def context_query(x: jax.Array) -> tuple[jax.Array, jax.Array | None]AttendsContext.context_keys
Section titled “AttendsContext.context_keys”def context_keys(context: jax.Array) -> tuple[jax.Array, jax.Array]AttendsContext.attend_context
Section titled “AttendsContext.attend_context”def attend_context( query: jax.Array, gate: jax.Array | None, key: jax.Array, value: jax.Array, *, lengths: jax.Array, queries: jax.Array | None = None, segments: jax.Array | None = None, max_segments: int | None = None, progress: jax.Array | None = None,) -> jax.ArrayContextEncoder
Section titled “ContextEncoder”How a context of token ids becomes the states a model’s cross-attention reads (encode_context):
table, its tokens through the model’s own token table, or backbone, the model’s own final states
over it in a pass of its own.
CrossAttention
Section titled “CrossAttention”class CrossAttention( layer_ids: tuple[int, ...], context_length: int, context_features: int | None = None, intermediate_size: int | None = None, activation: GatedActivation = 'swiglu',)The decoder layers that read a conditioning sequence, and its geometry.
Each layer in layer_ids runs the gated block before its own sublayers.
context_length is the most tokens a row’s context holds, which a decode
cache keeps the keys and values of. context_features is the width of
the context’s tokens; None reads the decoder’s own width. intermediate_size
is the block’s feed-forward width; None takes the width of the layer’s own
dense feed-forward. activation is the feed-forward’s gated activation,
IDEFICS’s SiLU unless named, whatever the decoder’s own feed-forward runs.
The attention has the decoder’s heads, key-value heads and head width, and
reads the context without positions.
CrossAttention.block
Section titled “CrossAttention.block”def block(decoder: CausalTransformer, index: int) -> Callable[..., GatedCrossAttention]Layer index’s gated block (CrossBlocks.block): the decoder’s norms, none under nGPT, a
gated MLP of the block’s own activation at its width (the layer’s own unless named), with nGPT’s
gains under nGPT, and the decoder’s attention heads, kernel, initializers and dtypes. A layer
with no feed-forward width to take needs intermediate_size.
CrossAttention.layer_key
Section titled “CrossAttention.layer_key”def layer_key(decoder: CausalTransformer, index: int) -> HashableThe width of layer index’s block’s feed-forward, all its block takes of the layer.
CrossAttention.flops
Section titled “CrossAttention.flops”def flops(decoder: CausalTransformer, index: int, lengths, context_lengths) -> floatLayer index’s block’s work (GatedCrossAttention.cross_flops), its gated norms’ maps
included as the layer routes (decoder_block.gate_flops), as its norms are built (block).
CrossAttention.held_flops
Section titled “CrossAttention.held_flops”def held_flops( decoder: CausalTransformer, index: int, lengths, context_lengths,) -> floatNone: the block’s matrices are its own, which train with it.
CrossBlocks
Section titled “CrossBlocks”class CrossBlocks(Protocol)A decoder’s cross-attention, which with_cross_attention takes: the layers that read a
conditioning sequence, the most tokens a row’s context holds, which a decode cache keeps the keys
and values of, the width of its tokens (None for the decoder’s own), and each layer’s block, built
from the decoder. A block runs before its layer’s own sublayers, reads the residual and the
context and joins the residual (GatedCrossAttention’s call); one that reads through its layer’s
own attention says so (reads_layer) and is handed it (LayerReads). CrossAttention (IDEFICS’s
gated block) and SharedCrossAttention (each layer’s own projections behind one gate) are two;
any conforming configuration is a third.
CrossBlocks.block
Section titled “CrossBlocks.block”def block(decoder: CausalTransformer, index: int) -> Callable[..., nn.Module]Layer index’s block factory, which takes its module name (and flax’s parent),
refusing a layer the block cannot read the context at.
CrossBlocks.layer_key
Section titled “CrossBlocks.layer_key”def layer_key(decoder: CausalTransformer, index: int) -> HashableWhat layer index’s block depends on beside the decoder’s fields and this configuration:
layers of equal specs and equal keys build one block, which a scanned run of them runs for each
(dew.nn.backbones.layer_plan.LayerSpec.cross).
CrossBlocks.flops
Section titled “CrossBlocks.flops”def flops(decoder: CausalTransformer, index: int, lengths, context_lengths) -> floatLayer index’s block’s forward work over rows of lengths tokens reading contexts of
context_lengths tokens.
CrossBlocks.held_flops
Section titled “CrossBlocks.held_flops”def held_flops( decoder: CausalTransformer, index: int, lengths, context_lengths,) -> floatThe part of flops done by matrices the decoder holds, its layer’s own, which train as its
trunk does: a frozen trunk’s take no weight gradient (dew.nn.flops.frozen_training_flops).
GatedCrossAttention
Section titled “GatedCrossAttention”class GatedCrossAttention( reads_layer: bool = False, norms: tuple[str, ...] = ('input_layernorm', 'post_attention_layernorm'), ngpt: bool = False, dropout_rate: float = 0.0, attention_impl: AttentionImpl = 'auto', force_fp32_for_softmax: bool = True, dtype: Dtype | None = None, precision: PrecisionLike = None,)One layer’s gated block: a residual with its gated attention over a
CrossContext and its gated feed-forward joined in.
norm and feedforward are the decoder’s factories, which take a name;
ngpt reads the stream unnormed and joins each output as nGPT’s step
(the module docstring), and norm is then None. The projections are
stored flat, [inputs, heads * head_dim], as the decoder’s own attention
stores them. attention_impl and force_fp32_for_softmax are the
decoder’s.
norms: tuple[str, ...]-
The norms the block builds by name (
norm): before its attention, and before its feed-forward.
GatedCrossAttention.attention
Section titled “GatedCrossAttention.attention”def attention(**parent={}) -> NormalAttentionThe block’s attention over its context: the attention every cross-attention runs
(NormalAttention), its projections flat under the decoder’s names, grouped over the
decoder’s key-value heads, keeping a cached call’s keys and values. parent is flax’s: None for
the unbound description its maps and work are read from.
GatedCrossAttention.projections
Section titled “GatedCrossAttention.projections”def projections() -> dict[str, tuple[int, int]]The attention’s maps as {name: (inputs, outputs)}.
GatedCrossAttention.cross_flops
Section titled “GatedCrossAttention.cross_flops”def cross_flops(lengths, context_lengths) -> floatThe block’s work over rows of lengths tokens reading contexts of
context_lengths: the attention’s (NormalAttention.cross_flops) and
the feed-forward’s per token.
LayerReads
Section titled “LayerReads”class LayerReads(NamedTuple)What a block that reads through its layer (reads_layer) is handed: the layer’s attention
input from the states a sublayer reads, normed as its attention reads it, its attention, and the
call’s progress (AttentionMetadata.progress), which that attention may read.
SharedCrossAttention
Section titled “SharedCrossAttention”class SharedCrossAttention(layer_ids: tuple[int, ...], context_length: int)Cross-attention through each named layer’s own attention: before its sublayers, the layer in
layer_ids reads the conditioning sequence with its own query, key, value and output projections
and norms, its logit scale, sinks and softcap, unrotated and uncausal
(AttendsContext), and the read joins the residual through one tanh gate the layer adds, zero
at the start, so the decoder computes its own function until the gates train. The only weights
it adds are the gates; the context’s tokens are the decoder’s width, which the layer’s key and
value projections read. context_length is the most tokens a row’s context holds, which a
decode cache keeps the keys and values of. A layer of no attention, or a KV-sharing one, projects
no keys of its own and is refused.
Its only new weights are the gates, so what it reads is what the layer’s own projections, trained
for its own sequence, make of the context: over a frozen trunk it learns no read those projections
do not already make, and a context the trunk must learn to read needs the trunk’s adapters (LoRA)
training beside it. A toy whose answer is the context’s first token (row 2320’s diagnosis, armada
IGGATE2) stays at the no-context loss under it whatever the rate, where IDEFICS’s block
(CrossAttention), fresh weights of its own, learns the read once they train at their own rate.
context_features: None-
None: the context is the decoder’s width, as each layer’s own projections read it.
SharedCrossAttention.block
Section titled “SharedCrossAttention.block”def block( decoder: CausalTransformer, index: int,) -> Callable[..., SharedGatedCrossAttention]Layer index’s gate over its own attention’s read (CrossBlocks.block), refused where
the layer’s mixer reads no context (AttendsContext).
SharedCrossAttention.layer_key
Section titled “SharedCrossAttention.layer_key”def layer_key(decoder: CausalTransformer, index: int) -> HashableNothing: a layer’s block takes its geometry from the layer’s own attention, which the layer’s spec builds, and reads through the attention the layer hands it.
SharedCrossAttention.flops
Section titled “SharedCrossAttention.flops”def flops(decoder: CausalTransformer, index: int, lengths, context_lengths) -> floatLayer index’s attention’s read of the context (AttendsContext.context_flops).
SharedCrossAttention.held_flops
Section titled “SharedCrossAttention.held_flops”def held_flops( decoder: CausalTransformer, index: int, lengths, context_lengths,) -> floatIts read’s projections, the layer’s own matrices (AttendsContext.context_projection_flops):
all of its work but the attention between them.
SharedGatedCrossAttention
Section titled “SharedGatedCrossAttention”class SharedGatedCrossAttention( reads_layer: bool = True, ngpt: bool = False, dropout_rate: float = 0.0, dtype: Dtype | None = None, precision: PrecisionLike = None,)One layer’s read of a conditioning sequence through its own attention
(SharedCrossAttention): the residual’s states normed as the layer’s attention reads them, its
query heads over them and its key and value heads over the context (AttendsContext), attended
and projected by the layer’s own weights, the read joined through one tanh gate,
alpha_cross_attn, zero at the start. heads is the layer’s key heads’ count and the widths of
a key and a value head, which a decode cache keeps the context’s keys and values at; a cached call
keeps them and each row’s count, so later steps read them without the context. ngpt joins the
read as nGPT’s step along the sphere (gated_join).
reads_layer: bool-
It reads the context through its layer’s own attention (
CrossBlocks,LayerReads).
context_mask
Section titled “context_mask”def context_mask( query: jax.Array, key: jax.Array, lengths: jax.Array, queries: jax.Array | None = None, segments: jax.Array | None = None,) -> jax.ArrayThe keys a read of a context sees (AttendsContext.attend_context): query [B, S, H, D] over a
context’s key [B, C, ..., D], each row’s first lengths keys real and, where given, each query
its own document’s keys alone (queries [B, S], segments [B, C]), unmasked by causality.
crossed_flops
Section titled “crossed_flops”def crossed_flops( decoder: CausalTransformer, lengths, context_lengths, *, held: bool = False,) -> floatdecoder’s cross-attention blocks’ work over rows of lengths tokens reading contexts of
context_lengths tokens (CrossBlocks.flops), or with held the part its own matrices do
(CrossBlocks.held_flops), each block on every pass of its layer (layer_passes): none without
cross-attention, and a call without a context runs none of it.
crossing
Section titled “crossing”def crossing(path) -> boolWhether a variables path runs through a cross-attention block: the filter that trains the
blocks alone (dew.batch.PathFilter).
encode_context
Section titled “encode_context”def encode_context( model: nn.Module, variables: Mapping, tokens: jax.Array, lengths: jax.Array, *, encoder: ContextEncoder, train: bool = False, rngs: dict[str, jax.Array] | None = None, segments: jax.Array | None = None, max_segments: int = 1, progress: Progress = END,) -> dict[str, jax.Array]The conditioning a model’s cross-attention layers read (CrossAttending) for contexts of
token ids: tokens [B, C], each row’s first lengths real, as encoder says, and where a row
packs the contexts of up to max_segments documents, each token’s document, segments [B, C]
(CONTEXT_SEGMENTS), which the backbone pass reads each of alone and the call’s own tokens of each
document read alone.
table gives each token’s row of the model’s own table at its stream’s width (stream_embeddings);
backbone the
model’s final states over each context alone (its hidden_states, or a native module’s own forward:
dew.nn.protocols.forward_method), its real tokens one document, with
the weights of the model’s own pass, an adapter’s included, at the run’s progress where the model
reads it (dew.nn.progress). Where the model trains nothing but its
cross-attention blocks (crosses_alone), which a pass without a context runs none of, that pass
trains nothing and its gradient is not taken. The states keep the dtype they are computed in, and
the model casts them to its blocks’ (context_dtype). variables are the model’s own, whole or
split from a frozen rest (dew.objectives.base.FROZEN). A model without cross-attention layers is
refused (TypeError), as is a backbone pass over one without final states; another encoder is
refused (ValueError).
gated_join
Section titled “gated_join”def gated_join( stream: jax.Array, branch: jax.Array, gate: jax.Array, *, ngpt: bool = False, precision: PrecisionLike = None,) -> jax.Arraystream with branch joined through gate, in at least float32 of the three and rounded once
to the stream’s dtype: added, broadcast over any axis the stream has beside the states’, or with
ngpt moved by the step of gate from the unit stream toward the unit branch, nGPT’s step on the
unit sphere, at the stream’s own length (_moved_on_the_sphere). The stream is scaled by its
largest element first, so no square of a finite stream overflows, and a zero gate leaves it bit for
bit, whatever its length: before a first layer the input embeddings are not unit.
sow_context_logits
Section titled “sow_context_logits”def sow_context_logits( module: nn.Module, query: jax.Array, key: jax.Array, lengths: jax.Array, queries: jax.Array | None = None, segments: jax.Array | None = None,) -> NoneA read of a context’s per-head largest logit over the keys it sees (context_mask), sown in
module’s qk collection (CONTEXT_LOGITS) where a caller opened it, so the layer’s own call takes
it in beside its own sequence’s (with_context_logits) and the QK-Clip bounds both.
with_context_logits
Section titled “with_context_logits”def with_context_logits(module: nn.Module, found: jax.Array) -> jax.Arrayfound, a layer’s own call’s per-head largest logits, beside those of a context the call read
first through the same projections (sow_context_logits).