dew.nn.backbones.layer_plan
The layer plan of a CausalTransformer: each layer’s kind, its spec, and which layers scan together.
A LayerKind names what a layer type builds: its mixer, and the attention
fields it overrides. A ResolvedKind is a kind with the model’s fields
filled in, and a LayerSpec is one layer’s plan. scan_groups finds the
runs of consecutive layers whose specs are equal, and nn.scan stacks each
run into one group.
| Name | Summary |
|---|---|
LayerKind | What the layers of one kind in the pattern do differently. |
LayerSpec | One layer’s resolved plan. |
ResolvedKind | One kind of layer with the model’s defaults filled in. |
LayerKind
Section titled “LayerKind”class LayerKind( window: int | None = None, bidirectional_window: bool = False, chunk: int | None = None, num_heads: int | None = None, num_kv_heads: int | None = None, rope_theta: float | None = None, rope_scaling: Scaling | None = None, head_dim: int | None = None, partial_rotary_factor: float | None = None, sinks: bool | None = None, qk_norm: bool | None = None, mixer: MixerBase | None = None,)What the layers of one kind in the pattern do differently.
The pattern names each layer’s kind, and this record says what the kind
means. A windowed kind is the “sliding attention” of the reference
configs. rope_theta and head_dim are the model’s unless the kind sets
its own. Rotary positions rotate every dimension of a windowed kind
unless it states its own partial_rotary_factor: Gemma 4 puts its
partial rotary on the global layers, and its sliding layers rotate every
dimension, where MiMo-V2-Flash rotates a third of every head.
mixer is the kind’s token mixer, a value from the mixers registry;
None uses the model’s mixer. A hybrid stack names its per-layer mixers
here, keyed by the kind names already in the pattern.
window: int | None-
The number of keys a causal layer of this kind attends to, its own included. None attends to all keys. A bidirectional layer reads its whole row, as DiffusionGemma’s decoder does, unless
bidirectional_window. bidirectional_window: bool-
Whether a bidirectional layer of this kind keeps
windowon both sides of a query, reading the keys within window - 1 positions of it, as ModernBERT’s local layers do (modeling_modernbert.py, |q - k| <= local_attention // 2). A causal layer ignores it. chunk: int | None-
The chunk size of chunked local attention. A layer of this kind reads only the keys at or before each query whose place has the same
place // chunkas the query’s, places counted over each document’s tokens (dew.nn.inputs.counted_places). This is MaxText’schunk_attn_window_sizeand Llama 4’sattention_chunk_size. None attends to all keys. A kind sets a window or a chunk, not both. num_heads: int | None-
This kind’s query head count; None takes the model’s. Laguna’s sliding layers have 64 where its full ones have 48 (num_attention_heads_per_layer).
num_kv_heads: int | None-
This kind’s key/value head count; None takes the model’s. Gemma 4’s global layers have fewer than its sliding ones (num_global_key_value_heads).
rope_scaling: Scaling | None-
This kind’s rope scaling (Llama 3.1’s, LongRoPE’s or YaRN’s), or its record. None uses the model’s. OLMo 3 scales only its full-attention layers by YaRN (configuration_olmo3.py:110-113).
partial_rotary_factor: float | None-
The fraction of this kind’s head the rotary turns; None takes the model’s on a kind that attends the whole sequence and every dimension on a windowed one.
sinks: bool | None-
Whether this kind’s attention adds a learned sink logit per head (
dew.nn.attention_sinks); None takes the model’sattention_sinks. MiMo-V2-Flash sinks its windowed layers alone. qk_norm: bool | None-
Whether this kind’s attention norms its queries and keys; None takes the model’s
qk_norm. Llama 4 norms its rotated layers alone (use_qk_norm, Llama4TextAttention). mixer: MixerBase | None-
This kind’s mixer value, or its record. None uses the model’s mixer.
LayerSpec
Section titled “LayerSpec”class LayerSpec( layer_type: str, kind: ResolvedKind, routed: bool, hash_routed: bool, width: int, sparsity: float, kv_shared: bool, provider: int | None, residual_site: ResidualSite | None, engram: int | None = None, prediction_target: int | None = None, attention_norm: bool = True, cross: tuple[Hashable] | None = None, dex: LayerDex | None = None,)One layer’s resolved plan.
The spec holds everything that the block’s parameters and computation depend on and that differs between layers. Two layers with equal specs have parameters of the same shapes and run the same program. A scan can therefore run them as iterations of one body, and a pipeline can run them at the same position of different stages. Settings of the whole model (norms, the attention settings, per-layer inputs, AltUp) are the same for every layer, so they are not repeated here.
engram: int | None-
The layer’s place among the engram layers, whose bucket ids it reads before its attention; None for a layer without a lookup.
prediction_target: int | None-
The layer’s own index when it is one of the DSpark drafter’s target layers, whose input streams’ mean it records under its own name (
CausalTransformer.draft_contextreadslayers_{index}), so it runs unrolled, as aproviderdoes; None for the rest. attention_norm: bool-
Whether the block norms its attention input; ModernBERT’s first block does not.
cross: tuple[Hashable] | None-
What the layer’s cross-attention block, which reads the call’s context first, depends on beside the decoder and its configuration, held in a tuple of one (
CrossBlocks.layer_key, whatever it is); None for a layer that reads none. Layers of equal specs build one block, which a scanned run of them runs for each. dex: LayerDex | None-
The layer’s differential extension on its attention’s chosen heads (
dew.nn.dex), or None. routed: bool-
Whether the feed-forward routes to the mixture’s experts.
hash_routed: bool-
Whether the routed feed-forward selects its experts by the token table.
width: int-
The dense feed-forward width, doubled on a sharing layer when the model asks for it.
sparsity: float-
The gaussian top-k fraction on the feed-forward gate, 0 for none.
kv_shared: bool-
Whether the layer reads its keys and values from an earlier layer.
provider: int | None-
The layer’s own index when a later layer reads what this layer leaves in the kv_store: its keys and values, or a CSA2 layer’s publications. Such a layer runs unrolled, because what it stores has to leave the stack’s loop. None for the other layers.
residual_site: ResidualSite | None-
The layer’s place among Kimi K3’s blocks of attention residuals, or None without them. It differs at every block boundary, so a scanned run never crosses one.
ResolvedKind
Section titled “ResolvedKind”class ResolvedKind( window: int | None, bidirectional_window: bool, chunk: int | None, num_heads: int, num_kv_heads: int, rope_theta: float, rope_scaling: Scaling | None, head_dim: int, partial_rotary_factor: float | None, sinks: bool, qk_norm: bool, mixer: MixerBase | None,)One kind of layer with the model’s defaults filled in.
LayerKind is what a config states, so a field the config leaves to the
model is None there. ResolvedKind is what the model resolved it to, so
num_heads, num_kv_heads, rope_theta and head_dim are numbers. The window, the
chunk and the rotary ramps stay optional, since a kind may have none.
mixer is passed through unchanged, and the model’s default applies when
it is unset.