Skip to content

dew.nn.mixers.attention

Grouped-query causal attention as a mixer kind.

The attention value builds this module: grouped-query projections, rotary positions, q/k norms and a fixed-size KV cache.

NameSummary
temperature_scaleLlama 4’s query multiplier at each absolute position (arXiv 2501.19399), log1p(floor((p + 1) / floor_scale)) * attn_scale + 1 in dtype, so the first floor_scale positions scale by 1.
exclusive_self_attentionRemove each head’s component along its token’s own value vector.
held_readsThe keys each row’s decode query reads from a paged cache: the row’s filled slots index [rows], or one where its query is padding (positions -1), as the dense path’s per-row key count reads.
mixed_attention_refusalWhy attention over ctx cannot run a serving step’s mixed call (dew.nn.inputs.Admitted), or None: it reads a full-precision cache, dense or one page pool, whole and causal, and nothing a row carries besides.
group_fieldsThe AttentionMetadata fields of groups a causal layer reads whole: image_groups where it reads images whole, block_groups where it reads blocks whole; a non-causal one reads none, since it sees its whole row already.
CallVisibilityA call’s own visibility over its keys (call_visibility), which replaces a layer’s causality, documents, groups and validity: its mask [B, 1, queries, keys], its key runs [B, queries, R, 2] (the template’s spans), the window the runs are read under, the places its keys are read at, a cached call’s queries’ places (None: the call’s own tokens, read at key_places), and its keys’ validity [B, keys] (AttentionMetadata.key_valid).
call_visibilityA call’s own visibility, None where it states none: its pairwise mask (AttentionMetadata.pairwise_mask), a windowed layer keeping each query to the keys within its window of the query’s rotary position where the call gives the keys’ positions; else its key runs (AttentionMetadata.key_runs), a windowed layer windowing each run over the key positions the call gives, its queries read at them too where they are its own tokens, or a cached call’s at its rotary positions through a mask.
dense_visibilityThe [B, 1, queries, keys] mask of a call’s visibility as a kernel reads it: mask with its documents, causality, window, block and image groups, key validity valid and key runs folded in (visibility_mask, the window counting real tokens where validity is given, or a runs call’s from its queries’ places over its keys’), joined with the layer’s chunks of chunk places, each query and key in the chunk of its counted place (places, key_places), never its cache slot.
GATE_ACTIVATIONSThe output gate’s activations (MixerContext.output_gate_activation).
ATTENTION_AXESThe logical axes of an attention layer’s projections, by name, as CausalSelfAttention declares them where it makes them; a packed projection (qkv_proj) declares none.
CausalSelfAttentionCausal self-attention with grouped-query heads, rotary positions, qk RMSNorm and a fixed-size KV cache.
AttentionMixerGrouped-query attention with optional image- and block-bidirectional masking and M-RoPE.
QSASelfAttentionCausalSelfAttention under QSA (qsa), which reads no pairwise mask or key runs: either may hide a key from one query that sees it from another, where QSA pools each key once (qsa_selection).
QSAAttentionMixerQwen4-Exp’s qwen_sparse_attention layer: the attention reading the keys its QSA indexer picks (dew.nn.qsa), the indexer’s geometry by the release’s config names.

function source

def temperature_scale(
positions,
floor_scale: float,
attn_scale: float,
*,
dtype: DTypeLike,
)

Llama 4’s query multiplier at each absolute position (arXiv 2501.19399), log1p(floor((p + 1) / floor_scale)) * attn_scale + 1 in dtype, so the first floor_scale positions scale by 1.

function source

def exclusive_self_attention(attention: jax.Array, value: jax.Array) -> jax.Array

Remove each head’s component along its token’s own value vector.

Exclusive self attention (arXiv 2603.09078) as lm-engine implements it (SoftmaxAttention._compute_xsa_output, softmax_attention/module.py at 45b6b57b): y - (<y, v> / <v, v>) v per token and query head, in fp32 at least, with the key/value head repeated over its query group. attention is [B, S, heads, D], value [B, S, kv_heads, D].

lm-engine divides by <v, v> with no guard, so a zero value vector (padding under a values norm, or a zero-initialised projection) is NaN there; here it leaves the output unchanged, which is the limit of the projection onto a vanishing direction’s span being empty.

function source

def held_reads(index: jax.Array, positions: jax.Array) -> jax.Array

The keys each row’s decode query reads from a paged cache: the row’s filled slots index [rows], or one where its query is padding (positions -1), as the dense path’s per-row key count reads. A freed row keeps its last request’s count until the next one is admitted, and the paged kernel reads every key under it: on an A100 at 128 slots and 32 requests a second, with 40 to 50 rows decoding, that was 5.1 ms of attention a step against cuDNN’s 2.0 over the dense cache.

function source

def mixed_attention_refusal(
ctx: MixerContext,
*,
alibi: bool,
bidirectional_images: bool,
bidirectional_blocks: bool,
) -> str | None

Why attention over ctx cannot run a serving step’s mixed call (dew.nn.inputs.Admitted), or None: it reads a full-precision cache, dense or one page pool, whole and causal, and nothing a row carries besides.

function source

def group_fields(causal: bool, *, images: bool, blocks: bool) -> tuple[str, ...]

The AttentionMetadata fields of groups a causal layer reads whole: image_groups where it reads images whole, block_groups where it reads blocks whole; a non-causal one reads none, since it sees its whole row already.

class source

class CallVisibility(NamedTuple)

A call’s own visibility over its keys (call_visibility), which replaces a layer’s causality, documents, groups and validity: its mask [B, 1, queries, keys], its key runs [B, queries, R, 2] (the template’s spans), the window the runs are read under, the places its keys are read at, a cached call’s queries’ places (None: the call’s own tokens, read at key_places), and its keys’ validity [B, keys] (AttentionMetadata.key_valid).

function source

def call_visibility(
metadata: AttentionMetadata,
rotary_positions,
*,
batch: int,
queries: int,
keys: int,
key_places,
sliding_window: int | None,
cached: bool = False,
) -> CallVisibility | None

A call’s own visibility, None where it states none: its pairwise mask (AttentionMetadata.pairwise_mask), a windowed layer keeping each query to the keys within its window of the query’s rotary position where the call gives the keys’ positions; else its key runs (AttentionMetadata.key_runs), a windowed layer windowing each run over the key positions the call gives, its queries read at them too where they are its own tokens, or a cached call’s at its rotary positions through a mask. key_places are where the keys are read otherwise.

function source

def dense_visibility(
mask,
documents,
causal: bool,
window: int | None,
valid,
runs,
places,
key_places,
queries: int,
keys: int,
chunk: int | None,
*,
block_groups=None,
image_groups=None,
) -> jax.Array

The [B, 1, queries, keys] mask of a call’s visibility as a kernel reads it: mask with its documents, causality, window, block and image groups, key validity valid and key runs folded in (visibility_mask, the window counting real tokens where validity is given, or a runs call’s from its queries’ places over its keys’), joined with the layer’s chunks of chunk places, each query and key in the chunk of its counted place (places, key_places), never its cache slot.

attribute source

GATE_ACTIVATIONS

The output gate’s activations (MixerContext.output_gate_activation).

attribute source

ATTENTION_AXES: dict[str, LogicalAxes] = {'q_proj': ('embed', 'heads'), 'k_proj': ('embed', 'kv'), 'v_proj': ('embed', 'kv'), 'gate_proj': ('embed', 'heads'), 'o_proj': ('attention', 'embed')}

The logical axes of an attention layer’s projections, by name, as CausalSelfAttention declares them where it makes them; a packed projection (qkv_proj) declares none.

Flax module source

class CausalSelfAttention(
reads: frozenset[str] = frozenset({'pairwise_mask', 'key_positions', 'key_runs', 'key_valid', 'position_bias', 'admitted'}),
qk_norm_weight: bool = True,
temperature_tuning: tuple[float, float] | None = None,
gate_projection: Literal['query', 'own', 'head'] = 'query',
bidirectional_images: bool = False,
bidirectional_blocks: bool = False,
mrope_section: tuple[int, int, int] | None = None,
rotary_axes: tuple[int, ...] | None = None,
rotary_pairs: Literal['half', 'adjacent'] = 'half',
rotary_interleaved: bool = False,
rotary_per_axis: bool = False,
qk_norm_after_rotary: bool = False,
packed: bool = False,
linear: Callable[..., Dense] = functools.partial(Dense, dot_general=float32_dot_general),
nope: bool = False,
alibi: bool = False,
exclusive_self_attention: bool = False,
qsa: QSA | None = None,
)

Causal self-attention with grouped-query heads, rotary positions, qk RMSNorm and a fixed-size KV cache.

decode=True runs against the cache: the first call writes the prompt and each later call appends, so prefill and decode are one path. Keys are rotated before they are cached, so rotary positions come from the cache index. causal=False is full attention, which a masked diffusion model reads its corrupted sequence with and an encoder reads its input with; with bidirectional_window a window keeps the keys within window - 1 positions on either side (dew.nn.visibility.window_sides). Decoding attends a bidirectional canvas over the frozen prefix a causal prefill cached, and writes nothing back. kv_shared marks a layer without K/V projections (Gemma 3n/4 cross-layer sharing): it reads the keys, values and places its provider stashed in kv_store, post rope and norm, and keeps no cache.

reads: frozenset[str]

Every MIXING field of its call (dew.nn.inputs.MixerCall): it masks by a pairwise mask or key runs, places keys, adds a position bias and runs a server’s mixed step.

temperature_tuning: tuple[float, float] | None

A (floor_scale, attn_scale) that scales each query by its position (temperature_scale), rotated or not: Llama 4’s unrotated layers’.

gate_projection: Literal['query', 'own', 'head']

Where the output gate comes from: the doubled query projection (Qwen3.5), a gate_proj of its own over the layer’s input (Muse Glimmer, modeling_muse_glimmer.py:305, 351), or such a gate_proj with one gate a head, which scales the head’s every value (LFM3, modeling_lfm3.py:229, 268-269). Either way the branch multiplies by its sigmoid.

bidirectional_images: bool

Whether a causal layer lets each token of an image (AttentionMetadata.image_groups) see the image’s later tokens too, within its window (Gemma 3’s, and Gemma 4’s sliding layers’).

bidirectional_blocks: bool

Whether a causal layer lets each token of a block (AttentionMetadata.block_groups) see the whole block, whatever its window (transformers’ block_sequence_ids): a decision prompt’s blocks, read both ways on the full layers that train so, a unified model’s media. A layer reading image groups as well sees the union.

rotary_axes: tuple[int, ...] | None

The channel widths of a rotary over several axes (dew.nn.rope.axis_tables), a vision tower’s patch grid, whose ids attention_metadata.rotary_positions carries [B, S, len(rotary_axes)]; None rotates by the positions.

rotary_pairs: Literal['half', 'adjacent']

The channels each angle turns (dew.nn.rope.rotate): the HF decoders’ halves, or the adjacent pairs of Llama 4’s vision tower.

rotary_interleaved: bool

Whether the rotary_axes take their angles in turn, pair by pair, where they otherwise each turn their own run of pairs (dew.nn.rope.axis_tables): MoonViT’s patch grid, a column’s pair beside a row’s.

rotary_per_axis: bool

Whether each of the rotary_axes turns its own channels in halves, Gemma 4’s vision rope (apply_multidimensional_rope), where Qwen 3.5’s turns the axes’ angles together over the whole head.

qk_norm_after_rotary: bool

Whether each head’s queries and keys are normed after their rotation, not before it: Hunyuan’s (modeling_hunyuan_v1_moe.py, HunYuanMoEV1Attention), a cached key normed as it is written. The query’s logit scale then multiplies after its norm, where a rotation folds it in.

packed: bool

Whether the queries, keys and values project through one kernel, qkv_proj, at init as from a checkpoint that stores them so (the Qwen 3.5 and DeepSeek-V4.1 vision towers).

linear: Callable[..., Dense]

The projections’ module: Gemma 4’s vision tower clips their inputs and outputs (dew.nn.vision.gemma4.Gemma4ClippableLinear).

nope: bool

No positional encoding: q and k enter the kernel unrotated, and the logits keep their scale (lm-engine’s position_embedding_type="nope").

alibi: bool

Linear key-position bias added to the logits, after any rotation of the queries and keys; BLOOM’s are unrotated (nope).

exclusive_self_attention: bool

XSA (arXiv 2603.09078): each head’s output loses its component along the token’s own value vector before the output projection.

qsa: QSA | None

Qwen4-Exp’s QSA (dew.nn.qsa): the keys each query reads are those its indexer picks among the ones it sees; None reads them all.

ctx: MixerContext

The layer’s geometry and attention policies, as the backbone resolves them for its kind (MixerContext).

reads_train: bool

Whether its call takes train (ReadsTrain): it drops attention probabilities while training.

value_dim: int

The width of one value head.

reads_context: bool

Whether it projects keys and values of its own to read a context with (context_keys): a KV-sharing layer does not.

def setup()
def model_flops(lengths) -> float

Query/output maps, unshared K/V maps and DEX’s maps of its chosen heads, plus QK and AV.

def parameter_flops(lengths) -> float

Learned QKV/output/gate maps, excluding data-dependent attention products.

def head_outputs() -> tuple[int, int]

Its heads’ count and each one’s output width, which DEX extends (dew.nn.dex.HeadOutputs).

def dex_fold_refusal() -> str | None

What its heads’ outputs pass through before o_proj, past DEX’s differential, which keeps the differential out of the projection (dew.nn.dex.folded); None where nothing does.

def projections() -> dict[str, tuple[int, int]]

The attention’s maps as {name: (inputs, outputs)}, keys and values absent where the layer shares them or the keys are the values, and values where routed experts give them (_value_expert_flops): what setup builds, the count reads and the packed group (_group) concatenates.

def projection_groups() -> tuple[ProjectionGroup, ...]

Its group (_group) where its variables hold it (ProjectionSites), which setup reads packed in the members’ place.

def context_heads() -> tuple[int, int, int]

Its key heads’ count and the widths of a key and a value head, as context_keys gives them.

def context_query(x)

Its query heads over x [B, S, E] as it reads another sequence (attend_context): projected and normed as its own call does them, scaled to the logit scale its checkpoint asks for, and placed nowhere: a context has no positions to rotate by, nor a temperature a place tunes. With the output gate, None where it gates none.

def context_keys(context)

Its key and value heads over a context [B, C, E], projected and normed as its own keys and values (_projected_kv), unrotated. A KV-sharing layer projects none of its own.

def attend_context(
query,
gate,
key,
value,
*,
lengths,
queries=None,
segments=None,
max_segments=None,
progress=None,
)

The layer’s own attention of query heads (context_query) over a context’s key and value heads (context_keys), each row’s first lengths keys real (none for a row without context, whose read its block leaves out) and, where given, each query reading its own document’s keys alone (queries [B, S], segments [B, C]): its unit sphere under nGPT, its sinks, logit softcap and kernel, unmasked by causality, then its output gate and projection. A context holds no token’s own value, so exclusive self attention excludes none.

def context_flops(lengths, context_lengths) -> float

Its read of a context’s work over rows of lengths tokens reading context_lengths (attend_context): its projections’ (context_projection_flops) and the attention between them.

CausalSelfAttention.context_projection_flops

Section titled “CausalSelfAttention.context_projection_flops”
def context_projection_flops(lengths, context_lengths) -> float

The part of its read of a context done by its own matrices: the query, gate and output projections over the rows of lengths tokens, the key and value projections over the context’s context_lengths.

dataclass source

class AttentionMixer(
bidirectional_images: bool = False,
bidirectional_blocks: bool = False,
mrope_section: tuple[int, int, int] | None = None,
nope: bool = False,
alibi: bool = False,
exclusive_self_attention: bool = False,
gate_projection: Literal['query', 'own', 'head'] = 'query',
qk_norm_weight: bool = True,
rotary_pairs: Literal['half', 'adjacent'] = 'half',
qk_norm_after_rotary: bool = False,
temperature_tuning: tuple[float, float] | None = None,
)

Grouped-query attention with optional image- and block-bidirectional masking and M-RoPE.

Geometry, norms and kernel policy come from the decoder context. Image and block bidirectionality, spatial rotary sections, NoPE and exclusive self attention are this mixer’s own fields, which a hybrid names on its attention kind ({"name": "attention", "fields": {"nope": true}}).

bidirectional_images: bool

Whether a causal layer reads an image whole (CausalSelfAttention.bidirectional_images).

bidirectional_blocks: bool

Whether a causal layer reads a block whole (CausalSelfAttention.bidirectional_blocks).

nope: bool

No positional encoding: q and k enter the kernel unrotated and the logits keep their scale (lm-engine’s position_embedding_type="nope", Granite 4.0-H).

alibi: bool

Per-head linear key-position bias (CausalSelfAttention.alibi).

exclusive_self_attention: bool

XSA (arXiv 2603.09078, lm-engine’s exclusive_self_attention): each head’s output loses its component along the token’s own value (exclusive_self_attention).

gate_projection: Literal['query', 'own', 'head']

Where the decoder’s output gate comes from (CausalSelfAttention.gate_projection).

qk_norm_weight: bool

Whether the decoder’s qk norm has a weight (CausalSelfAttention.qk_norm_weight): Muse Glimmer’s has none (modeling_muse_glimmer.py:307), nor Llama 4’s L2 norm.

rotary_pairs: Literal['half', 'adjacent']

The channels each rotary angle turns (CausalSelfAttention.rotary_pairs): halves, or adjacent pairs, as Llama 4’s rotated layers turn them.

qk_norm_after_rotary: bool

Whether each head’s queries and keys are normed after their rotation (CausalSelfAttention.qk_norm_after_rotary), as Hunyuan’s are.

temperature_tuning: tuple[float, float] | None

A (floor_scale, attn_scale) that scales each query by its position (temperature_scale, arXiv 2501.19399), as Llama 4’s unrotated layers do; None scales none.

def mixed_refusal(ctx: MixerContext) -> str | None
def build(ctx: MixerContext) -> Callable[..., nn.Module]

class source

class QSASelfAttention(CausalSelfAttention)

CausalSelfAttention under QSA (qsa), which reads no pairwise mask or key runs: either may hide a key from one query that sees it from another, where QSA pools each key once (qsa_selection).

dataclass source

class QSAAttentionMixer(
bidirectional_images: bool = False,
bidirectional_blocks: bool = False,
mrope_section: tuple[int, int, int] | None = None,
nope: bool = False,
alibi: bool = False,
exclusive_self_attention: bool = False,
gate_projection: Literal['query', 'own', 'head'] = 'query',
qk_norm_weight: bool = True,
rotary_pairs: Literal['half', 'adjacent'] = 'half',
qk_norm_after_rotary: bool = False,
temperature_tuning: tuple[float, float] | None = None,
indexer_n_heads: int = 4,
indexer_head_dim: int = 128,
indexer_budget: int = 2048,
indexer_compress_ratio: int = 4,
)

Qwen4-Exp’s qwen_sparse_attention layer: the attention reading the keys its QSA indexer picks (dew.nn.qsa), the indexer’s geometry by the release’s config names.

def mixed_refusal(ctx: MixerContext) -> str | None
def build(ctx: MixerContext) -> Callable[..., nn.Module]