Skip to content

dew.nn.backbones.layer_plan

The layer plan of a CausalTransformer: each layer’s kind, its spec, and which layers scan together.

A LayerKind names what a layer type builds: its mixer, and the attention fields it overrides. A ResolvedKind is a kind with the model’s fields filled in, and a LayerSpec is one layer’s plan. scan_groups finds the runs of consecutive layers whose specs are equal, and nn.scan stacks each run into one group.

NameSummary
LayerKindWhat the layers of one kind in the pattern do differently.
LayerSpecOne layer’s resolved plan.
ResolvedKindOne kind of layer with the model’s defaults filled in.

dataclass source

class LayerKind(
window: int | None = None,
bidirectional_window: bool = False,
chunk: int | None = None,
num_heads: int | None = None,
num_kv_heads: int | None = None,
rope_theta: float | None = None,
rope_scaling: Scaling | None = None,
head_dim: int | None = None,
partial_rotary_factor: float | None = None,
sinks: bool | None = None,
qk_norm: bool | None = None,
mixer: MixerBase | None = None,
)

What the layers of one kind in the pattern do differently.

The pattern names each layer’s kind, and this record says what the kind means. A windowed kind is the “sliding attention” of the reference configs. rope_theta and head_dim are the model’s unless the kind sets its own. Rotary positions rotate every dimension of a windowed kind unless it states its own partial_rotary_factor: Gemma 4 puts its partial rotary on the global layers, and its sliding layers rotate every dimension, where MiMo-V2-Flash rotates a third of every head.

mixer is the kind’s token mixer, a value from the mixers registry; None uses the model’s mixer. A hybrid stack names its per-layer mixers here, keyed by the kind names already in the pattern.

window: int | None

The number of keys a causal layer of this kind attends to, its own included. None attends to all keys. A bidirectional layer reads its whole row, as DiffusionGemma’s decoder does, unless bidirectional_window.

bidirectional_window: bool

Whether a bidirectional layer of this kind keeps window on both sides of a query, reading the keys within window - 1 positions of it, as ModernBERT’s local layers do (modeling_modernbert.py, |q - k| <= local_attention // 2). A causal layer ignores it.

chunk: int | None

The chunk size of chunked local attention. A layer of this kind reads only the keys at or before each query whose place has the same place // chunk as the query’s, places counted over each document’s tokens (dew.nn.inputs.counted_places). This is MaxText’s chunk_attn_window_size and Llama 4’s attention_chunk_size. None attends to all keys. A kind sets a window or a chunk, not both.

num_heads: int | None

This kind’s query head count; None takes the model’s. Laguna’s sliding layers have 64 where its full ones have 48 (num_attention_heads_per_layer).

num_kv_heads: int | None

This kind’s key/value head count; None takes the model’s. Gemma 4’s global layers have fewer than its sliding ones (num_global_key_value_heads).

rope_scaling: Scaling | None

This kind’s rope scaling (Llama 3.1’s, LongRoPE’s or YaRN’s), or its record. None uses the model’s. OLMo 3 scales only its full-attention layers by YaRN (configuration_olmo3.py:110-113).

partial_rotary_factor: float | None

The fraction of this kind’s head the rotary turns; None takes the model’s on a kind that attends the whole sequence and every dimension on a windowed one.

sinks: bool | None

Whether this kind’s attention adds a learned sink logit per head (dew.nn.attention_sinks); None takes the model’s attention_sinks. MiMo-V2-Flash sinks its windowed layers alone.

qk_norm: bool | None

Whether this kind’s attention norms its queries and keys; None takes the model’s qk_norm. Llama 4 norms its rotated layers alone (use_qk_norm, Llama4TextAttention).

mixer: MixerBase | None

This kind’s mixer value, or its record. None uses the model’s mixer.

dataclass source

class LayerSpec(
layer_type: str,
kind: ResolvedKind,
routed: bool,
hash_routed: bool,
width: int,
sparsity: float,
kv_shared: bool,
provider: int | None,
residual_site: ResidualSite | None,
engram: int | None = None,
prediction_target: int | None = None,
attention_norm: bool = True,
cross: tuple[Hashable] | None = None,
dex: LayerDex | None = None,
)

One layer’s resolved plan.

The spec holds everything that the block’s parameters and computation depend on and that differs between layers. Two layers with equal specs have parameters of the same shapes and run the same program. A scan can therefore run them as iterations of one body, and a pipeline can run them at the same position of different stages. Settings of the whole model (norms, the attention settings, per-layer inputs, AltUp) are the same for every layer, so they are not repeated here.

engram: int | None

The layer’s place among the engram layers, whose bucket ids it reads before its attention; None for a layer without a lookup.

prediction_target: int | None

The layer’s own index when it is one of the DSpark drafter’s target layers, whose input streams’ mean it records under its own name (CausalTransformer.draft_context reads layers_{index}), so it runs unrolled, as a provider does; None for the rest.

attention_norm: bool

Whether the block norms its attention input; ModernBERT’s first block does not.

cross: tuple[Hashable] | None

What the layer’s cross-attention block, which reads the call’s context first, depends on beside the decoder and its configuration, held in a tuple of one (CrossBlocks.layer_key, whatever it is); None for a layer that reads none. Layers of equal specs build one block, which a scanned run of them runs for each.

dex: LayerDex | None

The layer’s differential extension on its attention’s chosen heads (dew.nn.dex), or None.

routed: bool

Whether the feed-forward routes to the mixture’s experts.

hash_routed: bool

Whether the routed feed-forward selects its experts by the token table.

width: int

The dense feed-forward width, doubled on a sharing layer when the model asks for it.

sparsity: float

The gaussian top-k fraction on the feed-forward gate, 0 for none.

kv_shared: bool

Whether the layer reads its keys and values from an earlier layer.

provider: int | None

The layer’s own index when a later layer reads what this layer leaves in the kv_store: its keys and values, or a CSA2 layer’s publications. Such a layer runs unrolled, because what it stores has to leave the stack’s loop. None for the other layers.

residual_site: ResidualSite | None

The layer’s place among Kimi K3’s blocks of attention residuals, or None without them. It differs at every block boundary, so a scanned run never crosses one.

dataclass source

class ResolvedKind(
window: int | None,
bidirectional_window: bool,
chunk: int | None,
num_heads: int,
num_kv_heads: int,
rope_theta: float,
rope_scaling: Scaling | None,
head_dim: int,
partial_rotary_factor: float | None,
sinks: bool,
qk_norm: bool,
mixer: MixerBase | None,
)

One kind of layer with the model’s defaults filled in.

LayerKind is what a config states, so a field the config leaves to the model is None there. ResolvedKind is what the model resolved it to, so num_heads, num_kv_heads, rope_theta and head_dim are numbers. The window, the chunk and the rotary ramps stay optional, since a kind may have none. mixer is passed through unchanged, and the model’s default applies when it is unset.