Skip to content

dew.nn.cross_attention

Gated cross-attention from a conditioning sequence into a decoder’s layers.

Flamingo’s block (Alayrac et al. 2022, section 2.2) runs before each decoder layer it names. The layer’s residual first reads the conditioning sequence through attention, then runs a feed-forward, and each output joins the residual through a tanh gate:

x = x + tanh(alpha_cross_attn) * cross_attn(input_layernorm(x), context)
x = x + tanh(alpha_dense) * mlp(post_attention_layernorm(x))

Both gates start at zero, so a pretrained decoder with the block attached computes its own function until the gates train. The pretrained weights stay frozen while the block trains (dew.interop.Pretrained.with_cross_attention). The reference layer is IDEFICS’s IdeficsGatedCrossAttentionLayer (transformers’ modeling_idefics.py), and its names are the names used here. Its attention is the one every cross-attention runs, a diffusion block’s and Wan’s alike (dew.nn.attention_blocks.NormalAttention), held in the block’s own scope, over the decoder’s grouped key-value heads and keeping a cached call’s keys and values; the gates, the feed-forward and the residual join are the block’s.

A decoder call carries the sequence in its conditioning (dew.nn.inputs.CONTEXT), with each row’s count of real tokens (CONTEXT_LENGTHS). A row with no tokens adds nothing through either branch, so it computes the decoder’s own function. IDEFICS still runs the feed-forward for a token that attends to no image; here the block is absent for such a row, as it is for a call with no context at all. A cached call that carries the context keeps its keys and values in the cache, so later decode steps attend over them without it.

That is one form of a decoder’s cross-attention (CrossBlocks, which a decoder’s with_cross_attention takes): CrossAttention names it. SharedCrossAttention is another, which adds no attention of its own: each named layer reads the conditioning sequence through its own attention’s projections and norms, unrotated and uncausal, and joins the read through one tanh gate, zero at the start, the only weight it adds:

x = x + tanh(alpha_cross_attn) * self_attn.read(input_layernorm(x), context)

Any configuration that builds a conforming block per layer is a third.

A block reads and joins whatever residual its decoder carries: the plain stream (AltUp’s active copy among them), mHC’s streams, whose mean it reads and each of which its outputs join, Kimi K3’s depth state, whose mixture it reads and whose partial block its outputs join, and nGPT’s unit stream, which it reads unnormed and which each output joins as a step of its gate along the sphere, as nGPT’s own sublayers do.

A gate multiplies its branch as an outer product, so the gate’s gradient, the sum of the branch times its cotangent over every token, is a dot: XLA:CPU reduces a broadcast’s transpose four times further from float64 than torch’s cascade sum over 640 terms, and its dot as close as torch’s.

NameSummary
CONTEXT_LOGITSThe qk collection’s entry of each head’s largest logit over a context a layer’s own attention read (AttendsContext.attend_context), which the layer’s own maxima take in (with_context_logits).
CROSS_CACHE
CROSS_LENGTHSThe cache variable of each row’s real context tokens, [rows].
NAMEThe module name of a decoder layer’s cross-attention block, in every collection.
AttendsContextA layer’s own attention as a read of another sequence (SharedCrossAttention): whether it projects keys and values of its own to read one with, the key heads’ count and the widths of a key and a value head, its query heads over the layer’s input, its key and value heads over a context, its attention of the one over the other through its output projection, and that read’s work: dew.nn.mixers.attention.CausalSelfAttention’s, dew.nn.mla.MultiHeadLatentAttention’s (and so k-pool’s) and dew.nn.deepseek_v4.DeepseekV4Attention’s.
ContextEncoderHow a context of token ids becomes the states a model’s cross-attention reads (encode_context): table, its tokens through the model’s own token table, or backbone, the model’s own final states over it in a pass of its own.
CrossAttentionThe decoder layers that read a conditioning sequence, and its geometry.
CrossBlocksA decoder’s cross-attention, which with_cross_attention takes: the layers that read a conditioning sequence, the most tokens a row’s context holds, which a decode cache keeps the keys and values of, the width of its tokens (None for the decoder’s own), and each layer’s block, built from the decoder.
GatedCrossAttentionOne layer’s gated block: a residual with its gated attention over a CrossContext and its gated feed-forward joined in.
LayerReadsWhat a block that reads through its layer (reads_layer) is handed: the layer’s attention input from the states a sublayer reads, normed as its attention reads it, its attention, and the call’s progress (AttentionMetadata.progress), which that attention may read.
SharedCrossAttentionCross-attention through each named layer’s own attention: before its sublayers, the layer in layer_ids reads the conditioning sequence with its own query, key, value and output projections and norms, its logit scale, sinks and softcap, unrotated and uncausal (AttendsContext), and the read joins the residual through one tanh gate the layer adds, zero at the start, so the decoder computes its own function until the gates train.
SharedGatedCrossAttentionOne layer’s read of a conditioning sequence through its own attention (SharedCrossAttention): the residual’s states normed as the layer’s attention reads them, its query heads over them and its key and value heads over the context (AttendsContext), attended and projected by the layer’s own weights, the read joined through one tanh gate, alpha_cross_attn, zero at the start.
context_maskThe keys a read of a context sees (AttendsContext.attend_context): query [B, S, H, D] over a context’s key [B, C, ..., D], each row’s first lengths keys real and, where given, each query its own document’s keys alone (queries [B, S], segments [B, C]), unmasked by causality.
crossed_flopsdecoder’s cross-attention blocks’ work over rows of lengths tokens reading contexts of context_lengths tokens (CrossBlocks.flops), or with held the part its own matrices do (CrossBlocks.held_flops), each block on every pass of its layer (layer_passes): none without cross-attention, and a call without a context runs none of it.
crossingWhether a variables path runs through a cross-attention block: the filter that trains the blocks alone (dew.batch.PathFilter).
encode_contextThe conditioning a model’s cross-attention layers read (CrossAttending) for contexts of token ids: tokens [B, C], each row’s first lengths real, as encoder says, and where a row packs the contexts of up to max_segments documents, each token’s document, segments [B, C] (CONTEXT_SEGMENTS), which the backbone pass reads each of alone and the call’s own tokens of each document read alone.
gated_joinstream with branch joined through gate, in at least float32 of the three and rounded once to the stream’s dtype: added, broadcast over any axis the stream has beside the states’, or with ngpt moved by the step of gate from the unit stream toward the unit branch, nGPT’s step on the unit sphere, at the stream’s own length (_moved_on_the_sphere).
sow_context_logitsA read of a context’s per-head largest logit over the keys it sees (context_mask), sown in module’s qk collection (CONTEXT_LOGITS) where a caller opened it, so the layer’s own call takes it in beside its own sequence’s (with_context_logits) and the QK-Clip bounds both.
with_context_logitsfound, a layer’s own call’s per-head largest logits, beside those of a context the call read first through the same projections (sow_context_logits).

attribute source

CONTEXT_LOGITS = 'context_max_logits'

The qk collection’s entry of each head’s largest logit over a context a layer’s own attention read (AttendsContext.attend_context), which the layer’s own maxima take in (with_context_logits).

attribute source

CROSS_CACHE = frozenset({CROSS_KEY, CROSS_VALUE, CROSS_LENGTHS})

attribute source

CROSS_LENGTHS = 'cross_lengths'

The cache variable of each row’s real context tokens, [rows].

attribute source

NAME = 'cross_attention'

The module name of a decoder layer’s cross-attention block, in every collection.

class source

class AttendsContext(Protocol)

A layer’s own attention as a read of another sequence (SharedCrossAttention): whether it projects keys and values of its own to read one with, the key heads’ count and the widths of a key and a value head, its query heads over the layer’s input, its key and value heads over a context, its attention of the one over the other through its output projection, and that read’s work: dew.nn.mixers.attention.CausalSelfAttention’s, dew.nn.mla.MultiHeadLatentAttention’s (and so k-pool’s) and dew.nn.deepseek_v4.DeepseekV4Attention’s.

def context_heads() -> tuple[int, int, int]
def context_flops(lengths, context_lengths) -> float
def context_projection_flops(lengths, context_lengths) -> float
def context_query(x: jax.Array) -> tuple[jax.Array, jax.Array | None]
def context_keys(context: jax.Array) -> tuple[jax.Array, jax.Array]
def attend_context(
query: jax.Array,
gate: jax.Array | None,
key: jax.Array,
value: jax.Array,
*,
lengths: jax.Array,
queries: jax.Array | None = None,
segments: jax.Array | None = None,
max_segments: int | None = None,
progress: jax.Array | None = None,
) -> jax.Array

attribute source

How a context of token ids becomes the states a model’s cross-attention reads (encode_context): table, its tokens through the model’s own token table, or backbone, the model’s own final states over it in a pass of its own.

dataclass source

class CrossAttention(
layer_ids: tuple[int, ...],
context_length: int,
context_features: int | None = None,
intermediate_size: int | None = None,
activation: GatedActivation = 'swiglu',
)

The decoder layers that read a conditioning sequence, and its geometry.

Each layer in layer_ids runs the gated block before its own sublayers. context_length is the most tokens a row’s context holds, which a decode cache keeps the keys and values of. context_features is the width of the context’s tokens; None reads the decoder’s own width. intermediate_size is the block’s feed-forward width; None takes the width of the layer’s own dense feed-forward. activation is the feed-forward’s gated activation, IDEFICS’s SiLU unless named, whatever the decoder’s own feed-forward runs. The attention has the decoder’s heads, key-value heads and head width, and reads the context without positions.

def block(decoder: CausalTransformer, index: int) -> Callable[..., GatedCrossAttention]

Layer index’s gated block (CrossBlocks.block): the decoder’s norms, none under nGPT, a gated MLP of the block’s own activation at its width (the layer’s own unless named), with nGPT’s gains under nGPT, and the decoder’s attention heads, kernel, initializers and dtypes. A layer with no feed-forward width to take needs intermediate_size.

def layer_key(decoder: CausalTransformer, index: int) -> Hashable

The width of layer index’s block’s feed-forward, all its block takes of the layer.

def flops(decoder: CausalTransformer, index: int, lengths, context_lengths) -> float

Layer index’s block’s work (GatedCrossAttention.cross_flops), its gated norms’ maps included as the layer routes (decoder_block.gate_flops), as its norms are built (block).

def held_flops(
decoder: CausalTransformer,
index: int,
lengths,
context_lengths,
) -> float

None: the block’s matrices are its own, which train with it.

class source

class CrossBlocks(Protocol)

A decoder’s cross-attention, which with_cross_attention takes: the layers that read a conditioning sequence, the most tokens a row’s context holds, which a decode cache keeps the keys and values of, the width of its tokens (None for the decoder’s own), and each layer’s block, built from the decoder. A block runs before its layer’s own sublayers, reads the residual and the context and joins the residual (GatedCrossAttention’s call); one that reads through its layer’s own attention says so (reads_layer) and is handed it (LayerReads). CrossAttention (IDEFICS’s gated block) and SharedCrossAttention (each layer’s own projections behind one gate) are two; any conforming configuration is a third.

def block(decoder: CausalTransformer, index: int) -> Callable[..., nn.Module]

Layer index’s block factory, which takes its module name (and flax’s parent), refusing a layer the block cannot read the context at.

def layer_key(decoder: CausalTransformer, index: int) -> Hashable

What layer index’s block depends on beside the decoder’s fields and this configuration: layers of equal specs and equal keys build one block, which a scanned run of them runs for each (dew.nn.backbones.layer_plan.LayerSpec.cross).

def flops(decoder: CausalTransformer, index: int, lengths, context_lengths) -> float

Layer index’s block’s forward work over rows of lengths tokens reading contexts of context_lengths tokens.

def held_flops(
decoder: CausalTransformer,
index: int,
lengths,
context_lengths,
) -> float

The part of flops done by matrices the decoder holds, its layer’s own, which train as its trunk does: a frozen trunk’s take no weight gradient (dew.nn.flops.frozen_training_flops).

Flax module source

class GatedCrossAttention(
reads_layer: bool = False,
norms: tuple[str, ...] = ('input_layernorm', 'post_attention_layernorm'),
ngpt: bool = False,
dropout_rate: float = 0.0,
attention_impl: AttentionImpl = 'auto',
force_fp32_for_softmax: bool = True,
dtype: Dtype | None = None,
precision: PrecisionLike = None,
)

One layer’s gated block: a residual with its gated attention over a CrossContext and its gated feed-forward joined in.

norm and feedforward are the decoder’s factories, which take a name; ngpt reads the stream unnormed and joins each output as nGPT’s step (the module docstring), and norm is then None. The projections are stored flat, [inputs, heads * head_dim], as the decoder’s own attention stores them. attention_impl and force_fp32_for_softmax are the decoder’s.

norms: tuple[str, ...]

The norms the block builds by name (norm): before its attention, and before its feed-forward.

def attention(**parent={}) -> NormalAttention

The block’s attention over its context: the attention every cross-attention runs (NormalAttention), its projections flat under the decoder’s names, grouped over the decoder’s key-value heads, keeping a cached call’s keys and values. parent is flax’s: None for the unbound description its maps and work are read from.

def projections() -> dict[str, tuple[int, int]]

The attention’s maps as {name: (inputs, outputs)}.

def cross_flops(lengths, context_lengths) -> float

The block’s work over rows of lengths tokens reading contexts of context_lengths: the attention’s (NormalAttention.cross_flops) and the feed-forward’s per token.

class source

class LayerReads(NamedTuple)

What a block that reads through its layer (reads_layer) is handed: the layer’s attention input from the states a sublayer reads, normed as its attention reads it, its attention, and the call’s progress (AttentionMetadata.progress), which that attention may read.

dataclass source

class SharedCrossAttention(layer_ids: tuple[int, ...], context_length: int)

Cross-attention through each named layer’s own attention: before its sublayers, the layer in layer_ids reads the conditioning sequence with its own query, key, value and output projections and norms, its logit scale, sinks and softcap, unrotated and uncausal (AttendsContext), and the read joins the residual through one tanh gate the layer adds, zero at the start, so the decoder computes its own function until the gates train. The only weights it adds are the gates; the context’s tokens are the decoder’s width, which the layer’s key and value projections read. context_length is the most tokens a row’s context holds, which a decode cache keeps the keys and values of. A layer of no attention, or a KV-sharing one, projects no keys of its own and is refused.

Its only new weights are the gates, so what it reads is what the layer’s own projections, trained for its own sequence, make of the context: over a frozen trunk it learns no read those projections do not already make, and a context the trunk must learn to read needs the trunk’s adapters (LoRA) training beside it. A toy whose answer is the context’s first token (row 2320’s diagnosis, armada IGGATE2) stays at the no-context loss under it whatever the rate, where IDEFICS’s block (CrossAttention), fresh weights of its own, learns the read once they train at their own rate.

context_features: None

None: the context is the decoder’s width, as each layer’s own projections read it.

def block(
decoder: CausalTransformer,
index: int,
) -> Callable[..., SharedGatedCrossAttention]

Layer index’s gate over its own attention’s read (CrossBlocks.block), refused where the layer’s mixer reads no context (AttendsContext).

def layer_key(decoder: CausalTransformer, index: int) -> Hashable

Nothing: a layer’s block takes its geometry from the layer’s own attention, which the layer’s spec builds, and reads through the attention the layer hands it.

def flops(decoder: CausalTransformer, index: int, lengths, context_lengths) -> float

Layer index’s attention’s read of the context (AttendsContext.context_flops).

def held_flops(
decoder: CausalTransformer,
index: int,
lengths,
context_lengths,
) -> float

Its read’s projections, the layer’s own matrices (AttendsContext.context_projection_flops): all of its work but the attention between them.

Flax module source

class SharedGatedCrossAttention(
reads_layer: bool = True,
ngpt: bool = False,
dropout_rate: float = 0.0,
dtype: Dtype | None = None,
precision: PrecisionLike = None,
)

One layer’s read of a conditioning sequence through its own attention (SharedCrossAttention): the residual’s states normed as the layer’s attention reads them, its query heads over them and its key and value heads over the context (AttendsContext), attended and projected by the layer’s own weights, the read joined through one tanh gate, alpha_cross_attn, zero at the start. heads is the layer’s key heads’ count and the widths of a key and a value head, which a decode cache keeps the context’s keys and values at; a cached call keeps them and each row’s count, so later steps read them without the context. ngpt joins the read as nGPT’s step along the sphere (gated_join).

reads_layer: bool

It reads the context through its layer’s own attention (CrossBlocks, LayerReads).

function source

def context_mask(
query: jax.Array,
key: jax.Array,
lengths: jax.Array,
queries: jax.Array | None = None,
segments: jax.Array | None = None,
) -> jax.Array

The keys a read of a context sees (AttendsContext.attend_context): query [B, S, H, D] over a context’s key [B, C, ..., D], each row’s first lengths keys real and, where given, each query its own document’s keys alone (queries [B, S], segments [B, C]), unmasked by causality.

function source

def crossed_flops(
decoder: CausalTransformer,
lengths,
context_lengths,
*,
held: bool = False,
) -> float

decoder’s cross-attention blocks’ work over rows of lengths tokens reading contexts of context_lengths tokens (CrossBlocks.flops), or with held the part its own matrices do (CrossBlocks.held_flops), each block on every pass of its layer (layer_passes): none without cross-attention, and a call without a context runs none of it.

function source

def crossing(path) -> bool

Whether a variables path runs through a cross-attention block: the filter that trains the blocks alone (dew.batch.PathFilter).

function source

def encode_context(
model: nn.Module,
variables: Mapping,
tokens: jax.Array,
lengths: jax.Array,
*,
encoder: ContextEncoder,
train: bool = False,
rngs: dict[str, jax.Array] | None = None,
segments: jax.Array | None = None,
max_segments: int = 1,
progress: Progress = END,
) -> dict[str, jax.Array]

The conditioning a model’s cross-attention layers read (CrossAttending) for contexts of token ids: tokens [B, C], each row’s first lengths real, as encoder says, and where a row packs the contexts of up to max_segments documents, each token’s document, segments [B, C] (CONTEXT_SEGMENTS), which the backbone pass reads each of alone and the call’s own tokens of each document read alone.

table gives each token’s row of the model’s own table at its stream’s width (stream_embeddings); backbone the model’s final states over each context alone (its hidden_states, or a native module’s own forward: dew.nn.protocols.forward_method), its real tokens one document, with the weights of the model’s own pass, an adapter’s included, at the run’s progress where the model reads it (dew.nn.progress). Where the model trains nothing but its cross-attention blocks (crosses_alone), which a pass without a context runs none of, that pass trains nothing and its gradient is not taken. The states keep the dtype they are computed in, and the model casts them to its blocks’ (context_dtype). variables are the model’s own, whole or split from a frozen rest (dew.objectives.base.FROZEN). A model without cross-attention layers is refused (TypeError), as is a backbone pass over one without final states; another encoder is refused (ValueError).

function source

def gated_join(
stream: jax.Array,
branch: jax.Array,
gate: jax.Array,
*,
ngpt: bool = False,
precision: PrecisionLike = None,
) -> jax.Array

stream with branch joined through gate, in at least float32 of the three and rounded once to the stream’s dtype: added, broadcast over any axis the stream has beside the states’, or with ngpt moved by the step of gate from the unit stream toward the unit branch, nGPT’s step on the unit sphere, at the stream’s own length (_moved_on_the_sphere). The stream is scaled by its largest element first, so no square of a finite stream overflows, and a zero gate leaves it bit for bit, whatever its length: before a first layer the input embeddings are not unit.

function source

def sow_context_logits(
module: nn.Module,
query: jax.Array,
key: jax.Array,
lengths: jax.Array,
queries: jax.Array | None = None,
segments: jax.Array | None = None,
) -> None

A read of a context’s per-head largest logit over the keys it sees (context_mask), sown in module’s qk collection (CONTEXT_LOGITS) where a caller opened it, so the layer’s own call takes it in beside its own sequence’s (with_context_logits) and the QK-Clip bounds both.

function source

def with_context_logits(module: nn.Module, found: jax.Array) -> jax.Array

found, a layer’s own call’s per-head largest logits, beside those of a context the call read first through the same projections (sow_context_logits).