Skip to content

dew.nn.backbones.decoder_block

The decoder block that CausalTransformer stacks, and the parts it is built from.

The parts are the gated MLP, the mixture of experts, the block’s wiring and remat policies, and the multi-token prediction depth.

NameSummary
DECODER_REMATWhat a decoder recomputes in its backward pass when its step does not fit, weakest first; each rung is slower and holds less (docs/performance.md).
BlockWiringHow a block norms its two sublayers, and whether it scales its output.
DecoderBlockPre-norm decoder block: a token mixer, then a feed-forward, each added to the residual.
MTPBlockComputes one multi-token-prediction depth: the hidden states of the next depth.
MixtureThe experts that some layers route to, and how the router chooses among them.
RematPolicyWhat a recomputed block keeps for its backward pass, and where it keeps it.
remat_policyReturn value as the RematPolicy it names, or None.
remat_recordremat as a record stores it: a policy’s name in REMAT_POLICIES if it has one, its two lists otherwise, and None for no remat.

attribute source

DECODER_REMAT: tuple[RematPolicy | None, ...]

What a decoder recomputes in its backward pass when its step does not fit, weakest first; each rung is slower and holds less (docs/performance.md). On Qwen 3.5-0.8B at 8 x 3072 on an A100, ‘minimal’ cost 14.7% a token over none and kept 13.5 GiB of temporaries, ‘save_dot_except_mlp’ 18.4% and 6.4 GiB, none 59 GiB; ‘all_but_delta_memory’ recomputes a delta rule’s memory alone, as fla’s backward always does; ‘minimal_with_scan_and_conv’ keeps a recurrence’s scan output and its short conv output beside ‘minimal”s, so a step that does not fit recomputes the rest of a layer before its scan (on an A100 an 8-layer Mamba-2 decoder’s bf16 step over 16,384 tokens took 145.6 ms there and 149.4 ms at ‘minimal_with_conv’, for 1.1 GB more at its peak, docs/performance.md); ‘minimal_with_conv’ keeps the conv alone. A decoder without a recurrence skips both, as it would compile ‘minimal”s step. A decoder’s own remat is where it starts, and it moves up one rung at a time (CausalTransformer.recompute_more), past a rung that keeps nothing more of its layers’ names than the next, never past a policy the ladder names.

dataclass source

class BlockWiring(
pre_norms: bool = True,
output_norms: bool = False,
residual_norms: bool = False,
parallel_residual: bool = False,
shared_parallel_norm: bool = False,
layer_scalar: Literal['frozen', 'trainable'] | None = None,
branch_scale: float | None = None,
normed_residual: bool = False,
stream_multiplier: float = 1.0,
ngpt: NGPT | None = None,
)

How a block norms its two sublayers, and whether it scales its output.

pre_norms norms each sublayer’s input and output_norms norms its output. The input norms alone give the plain pre-norm block, both give Gemma’s sandwich block, and the output norms alone give OLMo 3’s post-norm block. residual_norms norms the stream itself after each sublayer joins it, the original Transformer’s post-LN block (OPT-350m’s), through the same two norms the pre-norm block reads its sublayers’ inputs with. ngpt gives the normalized transformer’s block, which has neither: each sublayer reads the unit residual and joins it by a learned step along the sphere (dew.nn.ngpt.stepped), attn_alpha and mlp_alpha its rates. normed_residual makes each sublayer’s normed input the stream its output joins, and stream_multiplier scales that stream as it does, the branch scaled by the model’s residual_multiplier: MiniMax-Text-01’s postnorm block and its alpha factors (modeling_minimax_text_01.py at a7351bf2, :1024-1058), x <- alpha norm(x) + beta f(norm(x)) a sublayer. The output norms are separate modules named for the sublayer outputs, so the input norms keep their names, and a checkpoint without output norms loads with two fewer leaves per layer. layer_scalar selects the reference’s frozen or trainable output scalar. One wiring applies to every layer, so it is not part of the per-layer specs that the scan groups by.

branch_scale: float | None

Initial value of learned per-feature attention and feed-forward residual scales.

Flax module source

class DecoderBlock(
norm_eps: float | None = 1e-05,
norm_type: Literal['rms', 'layer', 'fast_layer'] = 'rms',
norm_bias: bool = False,
norms: Mapping[str, NormSpec] | None = None,
scale_offset: bool = False,
scale_after_cast: bool = False,
per_layer_input_dim: int = 0,
gate_activation: GatedActivation = 'swiglu',
parallel: Callable[..., nn.Module] | None = None,
altup: AltUp | None = None,
laurel_rank: int | None = None,
hyper_connections: HyperConnections | None = None,
hash_routed: bool = False,
residual_multiplier: float = 1.0,
residual_site: ResidualSite | None = None,
routed: bool = False,
media_routed: bool = False,
engram: Callable[..., nn.Module] | None = None,
engram_index: int | None = None,
cross: Callable[..., nn.Module] | None = None,
prediction_target: bool = False,
prediction_site: Literal['input', 'output'] = 'input',
attention_norm: bool = True,
dropout_rate: float = 0.0,
remat: RematPolicy | None = None,
dtype: Dtype | None = None,
precision: PrecisionLike = None,
)

Pre-norm decoder block: a token mixer, then a feed-forward, each added to the residual.

mixer and feedforward are factories that take only a name. The mixer becomes self_attn, called (x, call) with one MixerCall, its tokens’ places and documents the call metadata’s Coordinates. The feed-forward becomes mlp and takes the normalized states, plus the token ids on a hash_routed block. A feedforward of None gives Mamba-2’s block, which has the mixer alone. wiring places the norms (BlockWiring).

The kv_store call argument passes one dict down the stack for KV-sharing mixers. per_layer_input is the layer’s slice of LayerInputs: its per-layer residual signal and, on a routed block, the replayed experts (dew.nn.protocols.Routes).

The residual takes one of four forms:

  • the plain stream;
  • altup, Gemma 3n’s [num_inputs, B, S, D] copies. The block predicts them, runs on the active copy and corrects them, then adds the per-layer residual to the copies past the first;
  • hyper_connections, mHC’s [B, S, hc_mult, D] streams with Sinkhorn-mixed residuals, passed as Carried(streams, pre) under Single-Pass;
  • residual_site, Kimi K3’s [B, S, blocks + 1, D] depth state.

laurel_rank adds LAuReL over the attention’s normed input; the sum with the attention residual is divided by sqrt(2).

engram first writes the layer’s n-gram lookup into the streams, reading the metadata’s engram_ids at engram_index. cross is the layer’s gated cross-attention factory (dew.nn.cross_attention): its block reads the metadata’s context and joins the residual, in its form, before the mixer reads it. A DSpark target layer (prediction_target) sows the mean of the streams its attention reads, or of those it hands on when prediction_site is ‘output’, as prediction_inputs/draft_context.

norm_eps: float | None

None: each RMS norm’s input dtype’s (RMSNorm.epsilon).

gate_activation: GatedActivation

The activation of the per-layer residual’s gated product. Gemma 3n/4 use the feed-forward’s activation here too.

parallel: Callable[..., nn.Module] | None

A branch summed with the feed-forward’s output before its output norm, called with the residual and that output (Gemma 4’s routed experts).

def model_flops(lengths) -> float

Compose the declared mixer and MLPs, independent of remat and storage.

def parameter_flops(lengths) -> float

The same submodules’ learned matrices, as PEFT’s frozen base work.

def projections() -> dict[str, tuple[int, int]]

The block’s own per-layer residual maps; its sublayers own theirs.

def setup()
def projection_groups() -> tuple[ProjectionGroup, ...]

The packed groups its mixer and its feed-forward declare (ProjectionSites).

Flax module source

class MTPBlock(
hyper_connections: HyperConnections | None = None,
norm_eps: float = 1e-05,
scale_offset: bool = False,
scale_after_cast: bool = False,
dropout_rate: float = 0.0,
remat: RematPolicy | None = None,
dtype: Dtype | None = None,
precision: PrecisionLike = None,
)

Computes one multi-token-prediction depth: the hidden states of the next depth.

enorm norms the token embeddings and hnorm norms the previous hidden states. The two are concatenated in that order, projected back to the model width, and passed through one decoder block. The released MTP weights were trained with this composition. With hyper_connections, the embedding and the raw residual streams are projected separately (e_proj, h_proj) and summed. Prediction steps may use a KV cache that is allocated separately from the trunk’s.

def projections() -> dict[str, tuple[int, int]]
def model_flops(lengths) -> float

Prediction input maps, one decoder block and any learned stream collapse.

def parameter_flops(lengths) -> float

Prediction input maps and the decoder’s learned matrices.

def setup()
def projection_groups() -> tuple[ProjectionGroup, ...]

The packed groups its block declares (ProjectionSites).

def states(
hidden,
embeds,
train: bool = False,
attention_metadata=None,
decode: bool = False,
prediction_phase: PredictionPhase = 'ordinary',
)

Return the normalized head input and the state that the next prediction step reads.

dataclass source

class Mixture(
experts: int,
top_k: int = 2,
layers: tuple[int, ...] | None = None,
score_function: str = 'softmax',
norm_topk_prob: bool = True,
scaling: float = 1.0,
groups: int = 1,
groups_per_token: int = 1,
group_score: str = 'top2',
bias: bool = False,
bias_in_weights: bool = False,
weight_sum_epsilon: float = WEIGHT_SUM_EPSILON,
weight_sum_min: float = 0.0,
router_softcap: float | None = None,
excluded_group_score: float | None = None,
routing_block: int | None = None,
block_experts: int | None = None,
scale_inputs: bool = False,
parallel: bool = False,
expert_features: int | None = None,
shared_features: int = 0,
shared_gate: bool = False,
implementation: str = 'auto',
dispatch: str = 'global',
capacity_factor: float | None = None,
hash_layers: tuple[int, ...] | None = None,
latent_features: int | None = None,
latent_norm: bool = False,
media_bias: bool = False,
)

The experts that some layers route to, and how the router chooses among them.

layers lists the sparse layers by index; a source’s cadence, such as Qwen3-MoE’s decoder_sparse_step, translates to this list. None makes every layer sparse (Mixtral). The routing fields are passed unchanged to Router, which documents them. parallel is Gemma 4’s placement (enable_moe_block): the experts run beside the dense feed-forward on the same residual, each output is normed, and the two are summed. That placement routes with Gemma4TextRouter, which refuses the routing fields.

expert_features is the experts’ width; None takes mlp_features. shared_features is the width of the one dense gated MLP that every token goes through, 0 for none (DeepseekV3MoE’s n_shared_experts times its expert width). shared_gate adds a learned per-token sigmoid to that shared branch (Qwen3.5 MoE). implementation names the kernel of moe.expert_projection. dispatch and capacity_factor are passed to moe.expert_dispatch; 'exchange' is expert parallelism over an expert mesh axis that divides the expert count. hash_layers route by DeepSeek V4’s token table, latent_features gives Kimi K3’s latent MoE, and media_bias is DeepSeek-V4.1’s image-span bias (SparseMLP, Router). bias_in_weights gates on the balancing bias as LLaDA-MoE does, and routing_block with block_experts is LLaDA2.2’s block routing. weight_sum_epsilon is the term a renormalization of non-softmax weights adds to their sum (moe.WEIGHT_SUM_EPSILON, DeepSeek’s; LFM2-MoE’s 1e-6), and weight_sum_min the least it divides by (ERNIE 4.5’s moe_norm_min, Router.weight_sum_min). router_softcap caps the gate logits (Laguna’s, Router.logit_softcap), and excluded_group_score is what an expert outside a token’s best groups is chosen by (Router.excluded_group_score).

beside_dense: bool

Whether its routed feed-forward runs beside a layer’s dense one, on the same residual, rather than in its place: Gemma 4’s parallel branch (build).

fixed_weights: bool

Whether a token’s expert weight is fixed whatever its router scores: one expert’s softmax score renormalized over itself (top_k=1 with norm_topk_prob, as a parallel or GPT OSS router does too), which is 1, so a model’s loss reaches the router through none of its outputs, only through a balance loss or a z-loss (dew.objectives.routing.refuse_fixed_router).

def generation_refusal() -> str | None

Why a generation loop cannot run the layers this mixture builds, or None.

Its exchange dispatch runs a shard_map, and a generation loop carries each draw’s device checks into the next step’s model call, which jax’s checkify cannot carry into one (jax-ml/jax#40907). A dew.inference.serving.Server step runs the model before it draws, so it serves the exchange; dispatch='global' computes the same layer. This refusal goes once a jax release carries the fix.

def router_scope(activation: GatedActivation | str) -> str

The scope its router’s score projection takes in its layer: GPT OSS’s (its activation ‘swigluoai’) and a parallel mixture’s (Gemma 4’s) router, any other’s SparseMLP.gate.

def build(
*,
out_features: int,
hidden_features: int,
activation: GatedActivation,
dense: Callable[..., nn.Module],
norm_eps: float,
scale_offset: bool = False,
scale_after_cast: bool = False,
swiglu_limit: float | None = None,
init_std: float | None = None,
output_init_std: float | None = None,
dtype: Dtype | None = None,
precision: PrecisionLike = None,
) -> Callable[..., nn.Module]

Build the routed feed-forward of decoder blocks and independent MLP mixers: GPT OSS’s experts and router under ‘swigluoai’ (GptOssMLP), Gemma 4’s branch beside the dense feed-forward where parallel (Gemma4Experts, beside_dense), else SparseMLP. The experts are expert_features wide, else hidden_features, the dense width. norm_eps, scale_offset and scale_after_cast are the block’s RMS norm, which a latent mixture’s output and Gemma 4’s branch take.

dataclass source

class RematPolicy(
save: tuple[str, ...] = (),
offload: tuple[str, ...] = (),
recompute: tuple[str, ...] = (),
)

What a recomputed block keeps for its backward pass, and where it keeps it.

A block under remat saves its inputs and recomputes its forward pass. save names the residuals (from RESIDUALS) to keep in device memory, and offload names those to move to pinned host memory. Both empty is MaxText’s full. recompute instead names the only residuals recomputed, each by the layer that computes it (a delta rule its memory, dew.nn.kernels.recurrence_chunk.RECOMPUTED), the block running without remat and keeping every other value; it takes neither list. A checkpoint policy cannot say it: one that saves anything but a name saves the value the name marks from the op that made it. REMAT_POLICIES holds MaxText’s recipes under this decoder’s names: q_proj… for MaxText’s query_proj…, and gate_proj/up_proj/down_proj for mlpwi_0/mlpwi_1/mlpwo. MaxText’s fused qkv_proj/mlpwi and AQT quantization have no counterpart here. A config gives a policy name or its lists.

def checkpoint_policy()

Return the policy that nn.remat runs the block under, or None to recompute everything.

function source

def remat_policy(
value: RematPolicy | str | Mapping[str, Sequence[str]] | None,
) -> RematPolicy | None

Return value as the RematPolicy it names, or None.

value is a RematPolicy, a name in REMAT_POLICIES, a record of save/offload names, or None for no recomputation at all. A config’s record arrives here untyped, so a value of any other kind raises ValueError and is not passed on.

function source

def remat_record(remat: RematPolicy | None) -> JSON

remat as a record stores it: a policy’s name in REMAT_POLICIES if it has one, its two lists otherwise, and None for no remat.