Skip to content

dew.nn.backbones.causal_transformer

The autoregressive transformer decoder that every language model in Dew trains.

By default the model has a token embedding, rotary positions, pre-norm blocks of grouped-query causal attention and a gated MLP, a final RMSNorm, and an fp32 head. Attention goes through the shared kernel path in dew.nn.attention, so a run picks reference/xla/cudnn/tpu the same way a diffusion run does, and decoding uses the same fixed-size KV cache helpers.

Parameter names follow the HF decoder layout: embed_tokens, layers_N.{input_layernorm, self_attn.{q,k,v,o}_proj, post_attention_layernorm, mlp.{gate,up,down}_proj}, norm, lm_head. Gemma’s two extra norms are the exception. HF calls them post_attention_layernorm and post_feedforward_layernorm even though they normalize sublayer outputs, so here they are attention_output_norm and mlp_output_norm, and the pre-norms keep their names. dew.interop.hf_decoders does that rename. A model family is supported only after its translator and a same-weight parity test against the reference have landed.

The block holds its token mixer in a slot. Any module called (x, call) -> x with one dew.nn.inputs.MixerCall, as CausalSelfAttention is, reading its tokens’ places and documents off the call metadata’s Coordinates and declaring the masks it reads (reads), can be self_attn without changes to the block, and that is where a linear-attention mixer goes.

NameSummary
CausalTransformerDecoder-only transformer over token ids: [B, S] int32 -> [B, S, vocab] fp32.

class source

class CausalTransformer(LinearProducts)

Decoder-only transformer over token ids: [B, S] int32 -> [B, S, vocab] fp32.

The defaults train a model from scratch: multi-head attention, swiglu, tied embeddings, no softcap. Every setting that differs between open decoders is a field here, so loading Qwen3 or Gemma3 is a mapping of config fields and needs no subclass. The field comments name the family that sets each one. A classic GPT block is norm_type='layer', norm_bias, an ungated mlp, mlp_bias and learned positions; the learned positions advance from each decode row’s count of its real tokens (DecodeCursor).

layer_types is the pattern, one kind per layer, and kinds says what each kind does. kv_shared_layers lists the layers that share by index: a trailing run of layers for Gemma 3n/4, or the layer after every indexer layer for GLM’s IndexShare. A sharing layer reads what the last earlier non-sharing layer of its kind stored: keys and values for attention, or the indexer’s selection for MLA.

When attention dropout is active the model runs the reference kernel, and it refuses an explicitly chosen fused kernel that cannot drop probabilities.

Interleaved mRoPE (Qwen3.5’s mrope_section) is partial_rotary_factor for text. With one position per token the three grids’ angles are equal, so text-only input reduces exactly to the partial rope; image-grid positions are not modelled. A mixer kind other than attention reads its own record, but the GQA geometry fields are still validated, so a translation has to fill them consistently.

MTP depth d pairs the previous depth’s state at position p with the embedding of the token at p + d, and scores the token that follows p + d (arXiv 2412.19437, section 2.2). Each depth is therefore one position shorter than the one before it.

With scan_layers, each run of like layers (layers whose resolved specs give the same parameter shapes and computation) runs as one scanned body, so compile time stops growing with depth. The variables tree stays the unscanned one (StackView). init always draws a run of like layers under the scan, once per run. When the mesh has a stage axis larger than one, the stack runs as a pipeline.

value_head_dim: int | None

The width of a value head; None is the query and key heads’ width. MiMo-V2-Flash attends 192-wide queries and keys over 128-wide values.

mlp_bias: bool

Whether both feed-forward projections have a bias: none by default, as Llama’s published mlp_bias: false, where Flax’s Dense adds one. The mlp activations gelu, gelu_exact and relu give an ungated feed-forward.

mlp_features: int | tuple[int, ...] | None

The dense feed-forward width. None means four times emb_features, a tuple gives one width per layer (Gemma 3n), and 0 means no feed-forward (Mamba-2).

mlp_multipliers: tuple[float, float] | None

The dense feed-forward’s gate and output multipliers (Falcon-H1’s mlp_multipliers, GatedMLP.gate_multiplier and down_multiplier); None for neither. The shared and routed experts take none.

context_length: int | None

The context a checkpoint states (max_position_embeddings), which an export writes back; None for max_seq_len. The model computes nothing from it: max_seq_len is its decode cache’s capacity, which a loader caps below a long context (decoder_parts.DEFAULT_MAX_SEQ_LEN).

position_embedding: Literal['rotary', 'learned', 'alibi']

How a layer whose kind names no mixer places its tokens (layer_mixer): rotary attention, or unrotated attention after a learned table the embedding adds (‘learned’) or with ALiBi’s biases (‘alibi’). A mixer a model or a kind names places its own tokens; the learned table is added whatever the mixers.

position_embedding_size: int | None

The number of rows in the learned position table; None means max_seq_len. It does not depend on the decode cache’s capacity.

position_embedding_offset: int

The number of reserved rows before learned position zero; OPT checkpoints have two.

rope_scaling: Scaling | None

The model’s rope scaling: Llama 3.1’s ramp, LongRoPE’s factors or YaRN’s record (rotary_freqs).

narrow_rotary_tables: bool

The rotary’s cos and sin round to the activations’ dtype before the fp32 rotation, as vLLM casts its cos/sin cache to the query’s dtype (MixerContext.narrow_rotary_tables); False keeps them at the rotation’s precision.

norm_eps: float

Llama 2’s and 3’s and Mistral’s published rms_norm_eps, where Flax’s RMSNorm takes 1e-6.

norms: Mapping[str, NormSpec] | None

The norms that differ from the model’s others, by module name (decoder_block.NORM_SITES): K2 Horizon’s grouped statistics, an epsilon or a weight of a norm’s own.

embedding_norm: bool

Normalize token embeddings before the first block, as BLOOM and ModernBERT do.

embedding_norm_reads: Literal['inputs', 'lookup']

What embedding_norm normalizes: every input embedding, a caller’s own and a media encoder’s among them, as BLOOM’s and ModernBERT’s embedding norms read their inputs_embeds (‘inputs’); or the table’s lookup alone, before media take their slots, as Muse Glimmer’s normed embedding module does (scaled_embeddings, ‘lookup’).

first_attention_norm: bool

Whether the first block norms its attention input. ModernBERT’s first block reads the embedding norm’s output as it is (modeling_modernbert.py, ModernBertEncoderLayer: attn_norm is Identity at layer 0).

embedding_features: int | None

The token embedding’s width where it differs from the stream’s (OPT-350m’s word_embed_proj_dim), bias-free project_in taking the lookup in and project_out the final states out to the head.

parallel_residual: bool

Whether attention and the feed-forward both read the same residual, as in GPT-NeoX.

shared_parallel_norm: bool

Both parallel branches read one LayerNorm, as in Phi, Falcon-7B and GPT-J.

attention_bias: bool

q/k/v biases, and o_proj’s unless o_proj_bias says: none by default, as Llama’s published attention_bias: false, where Flax’s Dense adds one.

value_scale: float | None

The factor that multiplies the values as they are projected, before they are cached: MiMo-V2-Flash’s attention_value_scale.

key_multiplier: float | None

The factor that multiplies the keys as they are projected, before they are rotated and cached: Falcon-H1’s key_multiplier.

embedding_multiplier: float

The factor that multiplies the token embeddings before the first layer: muP’s m_emb in lm-engine, and GraniteMoeHybrid’s embedding_multiplier.

residual_multiplier: float

The factor that multiplies every sublayer output before it is added to the residual stream: lm-engine’s m_residual, and GraniteMoeHybrid’s residual_multiplier.

residual_in_fp32: bool

Whether the residual stream between layers sums in at least fp32 whatever the compute dtype, as mamba_ssm’s residual_in_fp32 keeps it (transformers’ Mamba, Mamba2 and FalconMamba blocks): each sublayer reads the stream in the compute dtype and its output adds to the stream in fp32 (stack).

logits_scaling: float

The divisor of the logits: lm-engine’s m_width (lm_logits * (1 / m_width)), and GraniteMoeHybrid’s logits_scaling. The division is applied to the final states, in fp32, so every head that contracts them (output_table) gets the same logits that __call__ returns.

initializer_range: float | None

The std of lm-engine’s initialisation; None keeps each module’s own initializer.

When it is set, the embedding table (and an untied head) is drawn from N(0, initializer_range^2), and every hidden matrix (attention and Mamba-2 projections, conv taps, router, experts, dense MLPs) from N(0, (initializer_range / sqrt(logits_scaling))^2). Biases start at zero and norms at one. With logits_scaling set to m_width this is lm-engine’s init_method="mup", and with logits_scaling at 1 it is lm-engine’s "normal".

depth_scaled_init: bool

Whether, with initializer_range set, the projections back into the residual stream (o_proj, out_proj, down_proj) divide their std by sqrt(2 * num_layers). This is lm-engine’s use_depth_scaled_init.

ngpt: NGPT | None

The normalized transformer (dew.nn.ngpt.NGPT), NVIDIA’s Dory: every hidden state on the unit sphere, each block stepping its residual along it (BlockWiring.ngpt), the attention and feed-forward unit-norming their products and scaling them by learned gains, no norm with a weight anywhere, a unit norm between a loop’s passes, no norm before the head, and the logits times a learned per-token gain (logit_scale). None for none.

head_transform: Literal['gelu', 'gelu_exact'] | None

The activation of a BERT-style prediction head between the final norm and the vocabulary projection: a bias-free dense layer, this activation and a norm of the model’s kind (ModernBertPredictionHead, modeling_modernbert.py). None projects the final states directly. As in the feed-forward, ‘gelu’ is the tanh form and ‘gelu_exact’ the erf form. hidden_states stays the final norm’s output, the encoder’s.

head_bias: bool

A vocabulary bias added in fp32 after a tied or untied head’s product, as Phi’s, GPT-J’s and ModernBERT’s decoders add one (output_table’s bias).

embedding_zero_ids: tuple[int, ...]

Placeholder ids that the embedding lookup reads as token zero. The labels keep the original ids.

padding_idx: int | None

The id whose token embedding rows (embed_tokens, embed_tokens_per_layer) the lookup reads but never trains, as a source’s nn.Embedding(padding_idx=...) leaves them (Gemma 3n’s and Phi’s pad_token_id, TokenEmbedding.padding_idx); a tied head still trains the row. None trains every row.

kv_cache: KVCache

How attention layers store the decode cache: dense or paged, full or quantized (dew.nn.kv_cache). The parameters do not depend on it.

continuum: Levels | None

A continuum level on the final states before the head, or a chain of them (dew.nn.continuum): parameters an objective steps in context by its own loss (LMObjective(adaptation=...)), every head that reads the states, __call__’s and each prediction depth’s, reading the levels first. It starts as the identity. None gives the plain head.

causal: bool

Whether attention is causal. False gives full attention, the encoder that a masked diffusion language model denoises with. The parameter tree is the same either way.

per_layer_input_dim: int | None

The width of Gemma 3n/4 per-layer inputs, or None for a model without them. Each layer reads its own slice of an extra table and adds it to its input through its own gate.

mtp_layer_type: str | None

The layer kind the prediction depths build their mixer from. It need not occur in the trunk. None uses the full-attention kind when the pattern has one, and the first layer’s kind otherwise.

mtp_feedforward_type: str | None

The layer kind whose MLP mixer (MLPMixer) builds the prediction depths’ feed-forward, as Nemotron-H’s step pairs an attention block with a routed MLP block. None routes like the last layer, and is dense otherwise.

mtp_hyper_connections: HyperConnections | None

The residual streams of the prediction depths. None gives the depths plain residuals and normalized trunk inputs. Setting it opts the depth into streams, separately from the trunk’s hyper_connections.

attention_residuals: AttentionResiduals | None

Kimi K3’s attention residuals (dew.nn.attention_residuals): a softmax over finished blocks of layers that replaces the running residual. None disables them.

engram: Engram | None

DeepSeek-V4.1’s n-gram lookups or Qwen4-Exp’s per-layer embedding (dew.nn.engram) before each named layer’s attention, over the compressed vocabulary constants holds (engram_hashes/token_map).

dex: DifferentialExtension | None

The chosen heads whose attention outputs take a learned differential (dew.nn.dex), or None.

cross_attention: CrossBlocks | None

The layers that read a conditioning sequence through Flamingo’s gated cross-attention (dew.nn.cross_attention), before their own sublayers; None for a model without them. A call passes the sequence as its conditioning.

dspark: DSpark | None

DeepSeek’s block drafter (dew.nn.dspark, V4.1 and V4-Flash-0731), or None for a model without one. Its stages are decoder blocks that attend through a DSparkAttention built from the V4 attention of its layer_type kind. Its target layers sow the stream means that the drafter reads (draft_context).

activation_sparsity_pattern: tuple[float, ...] | None

Gemma 3n’s gaussian top-k activation sparsity, one fraction per layer.

mask_token_id: int | None

The vocabulary id that a masked-diffusion objective corrupts tokens to. None means plain training.

shifted_logits: bool

Whether masked diffusion reads position i’s prediction from position i-1’s logits, as a checkpoint continued from an autoregressive one predicts the token after each position (Dream-v0); False reads each position’s own (LLaDA). The forward ignores it: it says how the checkpoint’s logits line up, which dew.diffusion.discrete.aligned applies for the masked loss and the sampler alike.

scan_layers: bool

Runs of like layers under flax’s scan; a decode step over resident weights scans a period’s repeats as one body too (scan_groups(periodic=True)), so its program grows with the period, not the depth.

bank_layers: int | None

The most layers one scanned run holds, which is also how many layers its parameter bank stacks. A longer run of like layers is split into consecutive runs of at most this many, each with its own bank under its own name. None keeps a whole run in one bank. scan_layers and init read it, and because init draws each run under one scan, it also changes what init draws for a given seed. The split bounds the memory needed to build a host-resident bank and read it back, so set it for a deep stack that is offloaded to the host.

loop: Loop | None

The layer stack run several times over its own output, its parameters shared (Loop), as Ouro’s looped decoder runs it. None runs it once.

remat: RematPolicy | None

The rematerialization policy. Each block is recomputed in the backward pass, keeping its inputs, any K/V passed to later layers, and the residuals the policy names. A config gives a name from REMAT_POLICIES or a record of residual names to save or offload. None recomputes nothing. Init and cached decode call the blocks directly, and the stored parameters have the same layout either way.

layer_collections: tuple[str, ...]

The collections a reader of layer states opens (LayerStates).

wiring: BlockWiring

How every block wires its two sublayers (BlockWiring), from the model’s own dials, each a field of its own (docs/design/api.md): the one value the stack’s blocks, the prediction depths, DSpark’s stages and the FLOP count build with.

init_stds: tuple[float | None, float | None]

The normal std of the hidden matrices and of the projections back into the residual stream. Both are None when initializer_range is None, and the modules then use their own initializers.

position_rows: int

The rows of the learned position table (position_embedding_size, else max_seq_len).

position_capacity: int | None

The places a row’s learned positions cover, the table’s rows past its reserved offset; None for rotary or ALiBi positions, which cover any (PositionCapacity).

mlp_widths: tuple[int, ...]

Each layer’s dense feed-forward width.

hidden_features: int

The model’s single feed-forward width. The routed experts fall back to it and the prediction depths use it. A model whose layers have different widths has none and raises ValueError.

hash_layers: set

The sparse layers that route by the mixture’s token table.

sparse_layers: tuple[int, ...]

The layers whose feed-forward routes to experts.

sharing_layers: tuple[int, ...]

The sorted indices of the layers that read what another layer stored.

kv_sharing: dict

A map from each sharing layer’s index to the provider layer it reads.

A sharing layer has no K/V of its own (and, for MLA, no indexer). It reads the last earlier non-sharing layer of its own type. The map is empty unless kv_shared_layers is set, and a sharing layer with no earlier provider raises ValueError.

in_context: bool

Whether the model holds a continuum level (SteppedInContext).

in_context_levels: int

The continuum levels the model holds, a chain’s count (SteppedInContext).

embedding_factor: float

The factor scaled_embeddings scales an embedding by, unrounded, for a mixture of the table’s rows to take in its own arithmetic (dew.nn.diffusion_gemma.soft_embeddings).

context_width: int

The width of a context’s tokens (CrossAttending): the spec’s, or the decoder’s own.

context_dtype: jnp.dtype

The dtype a context’s tokens enter in (CrossAttending): the decoder’s compute dtype, at least float32, as its projections widen what they read.

cache_rebuild_position: int | None

Where LongRoPE’s factors switch from short to long, past which cached keys go stale: the original length of every trunk layer’s LongRoPE, its kind’s or the model’s (kind_of), None where no layer takes LongRoPE. Kinds that switch at different lengths are refused, since a cache rebuilds at one.

layer_mixer: MixerBase

The mixer a layer whose kind names none builds: the model’s own, else attention rotated or not by its position embedding.

declared_mixers: tuple[MixerBase, ...]

The mixer values it names: its own, then each layer kind’s.

def kind_of(layer_type: str) -> ResolvedKind

Return the ResolvedKind of layer_type, with the model’s defaults where the kind is unset. A kind’s rope scaling takes the model’s place.

def mixer_context(
kind: ResolvedKind,
layer_type: str,
kv_shared: bool,
dex: LayerDex | None = None,
dex_start: DexStart | None = None,
routed: bool = False,
) -> MixerContext

Return one layer’s mixer geometry as a MixerContext built from the resolved kind.

head_dim, rope_theta, window, the rope scaling, the partial rotary, the sinks and the query-key norm take the layer kind’s resolved values (kind_of), and the precision the model’s layers take (products). Every other field is the model’s field of the same name. A kind builds its DecoderBlock factory from this context and its own record, and setup chooses the mixer there and nowhere else.

def bank_sites(store: Variables) -> tuple[DecoderBank, ...]

Where this decoder’s layer stack is stored in store, the variables its banks are drawn from: scanned runs, or one layer per bank.

groups already records which of the two it is. A scanned stack declares its runs and a plain loop declares one-layer runs, and run_stack fetches a one-layer run the same way it fetches a longer one. A run ends where the store’s rows differ (StackView.layer_structures: layers a checkpoint quantizes differently, or adapts in part), so an unbound decoder plans its runs from the store’s shapes as a decoder bound to them reads them. scanned records whether those fetches are sequenced, which a host layout requires.

def stored_view(store: Variables) -> StackView

How store holds this decoder’s layer stack (bank_sites), as a decoder bound to its shapes plans its runs (SteppedFeedForward).

def layer_kinds(types: Sequence[str]) -> dict[str, ResolvedKind]

Resolve every kind the pattern names, plus the kinds of the prediction depths and the drafter.

Raises ValueError when the pattern does not have one kind per layer, or when kinds describes a kind that no layer has.

def model_flops(lengths) -> float

Real-row hidden-state work, summing the same block factories as setup (decoder_counts). A context’s cross-attention work is cross_flops, which needs the context’s lengths.

def parameter_flops(lengths) -> float

Learned matrices, separately from nonparameter attention/state work.

def projections() -> dict[str, tuple[int, int]]

The trunk’s own maps as {name: (inputs, outputs)}, the head included: what setup builds and the counts read.

def counted_blocks() -> list[DecoderBlock]

Each layer’s block as setup builds it, unbound, which its counts read (decoder_counts).

def prediction_blocks() -> list[MTPBlock]

Each prediction depth’s block as setup builds it, unbound, which its counts read.

def replicated_flops(stages: int, lengths) -> float

The work of the layers a pipeline of stages runs on every stage rather than once (stage_split): those before and after the run its stages repeat.

def layer_passes(index: int) -> int

How many times a call runs layer index: a loop’s steps for a layer it repeats, else once.

def logits_flops(lengths) -> float

The vocabulary matrix and optional prediction transform, project_out’s included. A tied head reads the embedding table, vocab_size rows of the embedding’s width.

def prediction_flops(lengths, *, prediction_lengths=None) -> float

Each trained prediction depth shortens the real sequence by one token.

CausalTransformer.prediction_parameter_flops

Section titled “CausalTransformer.prediction_parameter_flops”
def prediction_parameter_flops(lengths, *, prediction_lengths=None) -> float

The same shortened depth forwards’ learned matrices, head included.

def prediction_matrix_usage(path: tuple[str, ...], lengths, *, prediction_lengths=None)

Each depth’s maps run once; the shared head runs on every shorter state.

def matrix_usage(path: tuple[str, ...], lengths, *, logits: bool = False)

Native matrix scopes declare head versus trunk work and repeated loop passes.

def setup()

Build the embeddings, the layers, the prediction depths mtp, the final norm, and lm_head when the embeddings are not tied.

Each layer has one LayerSpec, is built by block, and is scanned in groups.

def mixer_of(index: int) -> nn.Module

Layer index’s mixer, unbound, as its kind builds it over the layer’s context.

def drawn_like(index: int) -> Hashable

What layer index draws as: its scan key and what its extension starts from (_dex_start), so like layers draw under one scan. A method interceptor that gives a layer parameters of its own (an adapter’s factors, dew.lora) keys it apart where they differ.

def cross_flops(lengths, context_lengths) -> float

The cross-attention blocks’ work, which model_flops leaves out (crossed_flops).

def held_cross_flops(lengths, context_lengths) -> float

The part of cross_flops its own matrices do, which train as its trunk does (crossed_flops).

def states_and_logits(tokens, **kwargs={})

Return the prediction-input states and the logits from one forward pass.

A speculative decoder verifies with both. The logits give the target distribution, and the states seed the next block’s prediction depths. The states are the normalized states, except for V4, where they are the uncollapsed residual streams.

def states_and_logits_at(tokens, slots, **kwargs={})

Return the prediction-input states and the logits at one slot per row, or at [rows, K] slots.

A prefill needs logits only at slots, the positions the first draw reads. The head over every prompt position would be the largest array the forward allocates, [rows, width, vocab], and a decoder keeps only one row of it. Gathering the states before the head cuts the head’s work to [rows, features].

def head_bounds(states) -> HeadTerms

What bounds a slot loss’s curvature and slope in the states head_logits reads (HeadTerms): the bound rows (vocabulary_rows), softcap and transform, its norm read at the states (BoundedNorm), each kernel as Qwix’s rounding can grow it (_head_growth).

def in_context_level(variables)

The continuum level’s parameters in variables, None without a level (SteppedInContext).

def with_in_context_level(variables, level)

variables holding level as the continuum level’s parameters (SteppedInContext).

def continuum_states(hidden, index: int | None = None)

Final states hidden past the continuum levels, the head’s input, or past level index alone (chain_states), read at the levels’ own parameters (SteppedInContext).

def level_terms(hidden, parts: tuple[str, ...], index: int = 0, passed: bool = False)

What bounds a damped step of continuum level index’s parts at its states hidden (Continuum.step_terms), its maps’ inputs as a checkpoint’s quantizers round them.

def head_logits(x)

The fp32 head over states past the continuum levels (continuum_states), or final states.

def mtp_hidden_states(
hidden,
tokens,
train: bool = False,
input_embeddings=None,
image_groups=None,
block_groups=None,
rotary_positions=None,
coordinates: Coordinates | None = None,
conditioning: Mapping[str, jax.Array] | None = None,
)

Return one final-normed state array per shifted prediction depth.

Each depth combines the preceding hidden state with the next token’s embedding, including any media replacement, and uses the next token’s positions. A pair counts only when both of its positions are valid and belong to the same packed document, so padded positions in between never become keys. A sequence no longer than the number of depths raises ValueError. The depths keep no cross-attention: hidden, the trunk’s states, already read the call’s conditioning, which they take as the trunk does and leave unread.

def mtp_logits(
hidden,
tokens,
train: bool = False,
input_embeddings=None,
image_groups=None,
block_groups=None,
rotary_positions=None,
coordinates: Coordinates | None = None,
conditioning: Mapping[str, jax.Array] | None = None,
)

Return the shared language head’s logits for each prediction depth’s hidden states; a call’s conditioning is the trunk’s (mtp_hidden_states).

def mtp_step(
hidden,
tokens,
*,
depth: int = 0,
input_embeddings=None,
rotary_positions=None,
decode: bool = False,
prediction_phase: PredictionPhase = 'ordinary',
coordinates: Coordinates | None = None,
)

Run one unshifted prediction step, optionally appending to the depth’s own KV cache.

Call init_mtp_cache before cached steps. hidden is the target model’s preceding state, and tokens or input_embeddings give the candidate next token, as in vLLM’s Qwen3_5MultiTokenPredictor. Returns the step’s logits and hidden state; a chained draft’s next step reads that state in place of the target’s. With index_share_for_mtp_iteration, a cached “extend” step publishes its index selections and a “draft” step reuses them, while “ordinary” always recomputes them. Its tokens are laid out by their coordinates.

def token_embeddings(tokens)

Return the embeddings a prediction depth pairs with tokens.

mtp_hidden_states reads the unscaled table, so this method does that lookup, with embedding_zero_ids read as token zero, and nothing else. A drawn token is always text, so no media embedding replaces it.

def stream_embeddings(tokens)

token_embeddings at the stream’s width, through project_in past an embedding’s own: what a prediction depth pairs with tokens, and a context’s tokens a cross-attention reads by the table.

def scaled_embeddings(x)

Return the token embeddings x scaled the way the first layer reads them.

x is normalized first where the norm reads the lookup alone (embedding_norm_reads, Muse Glimmer), multiplied by sqrt(emb_features) when embedding_scale is set (Gemma), then by embedding_multiplier (lm-engine, GraniteMoeHybrid). The decoder, the multimodal wrapper and the Qwen-Image conditioner all scale here.

Gemma casts embed_scale to the embedding weight dtype. The lookup holds that table in fp32, so the factor stays fp32 and only the product is rounded (a bf16 factor would be 34.0 at hidden size 1152, not 33.941…). lm-engine multiplies in fp32, and scaled does the same.

def draft(
context,
tokens,
*,
decode: bool = True,
coordinates: Coordinates | None = None,
choose=None,
)

Draft a DSpark block after each row’s last context position.

context [B, M, targets * D] comes from draft_context (the stream means the target layers sow), its coordinates mark its real positions (real_slots: its validity less a padding document, every one where None; one document a row, as a draft continues one), and tokens [B] are the tokens drawn after it. Returns (ids [B, block + 1], logits [B, block, vocab], confidence [B, block]). Tokens are chosen by choose(index, logits), or greedily when choose is None (dew.nn.dspark.draft). For a cached draft, call init_draft_cache first. With tokens None the call only appends context, and with context None it drafts after what the windows already hold. A model without a drafter raises ValueError.

def reach_drafter(tokens, dtype)

Create the DSpark drafter’s parameters, which no forward pass reaches.

It drafts after tokens over a zero context as wide as the drafter’s first stage projects.

def draft_context(prediction_inputs: Mapping) -> jax.Array

Return DSpark’s context [B, S, targets * D] from prediction_inputs.

A forward pass sows prediction_inputs when the caller makes that collection mutable. The context is each target layer’s stream mean, concatenated in target_layers order.

def init_draft_cache(batch_size: int)

Allocate the drafter’s window caches, separately from the trunk’s cache.

def init_mtp_cache(batch_size: int)

Allocate the prediction depths’ caches, separately from the trunk’s cache.

def hidden_states(tokens, **kwargs={})

Return the final normalized states, without the vocabulary projection.

def capture_layers(module: nn.Module, method: str) -> bool

The capture_intermediates filter that keeps every layer’s output (LayerStates).

It matches the __call__ of the stack’s layers_N modules, a scanned run’s layers_3_7 and a pipeline stage’s layers_j included, and StackView.unstack lays what those sow out per layer, as the plain loop does; layer_state reads one layer back. Neither is a Flax method, so the filter the capture calls on every module runs no module call.

def layer_state(collections: Mapping[str, Mapping], index: int) -> jax.Array

Layer index’s output out of what the forward kept (LayerStates): the last layer’s as the stack returns it, and another’s as capture_layers kept it, the last call’s, a looped layer’s last pass; a model that carries several residual copies (AltUp’s) returns them all at every layer. index counts as a Python index does, -1 the last layer.

def logits(tokens, *, train: bool = False, **fields={})

Return __call__’s fp32 logits over hidden_states(tokens, **fields), which write no cache unless fields ask to decode.

def logits_from_hidden(hidden)

Return the logits of final states hidden: the head __call__ and every MTP depth score with.

def hidden_and_mtp_inputs(
tokens,
train: bool = False,
decode: bool = False,
input_embeddings=None,
image_groups=None,
block_groups=None,
rotary_positions=None,
attention_pairwise_mask=None,
attention_key_positions=None,
attention_key_runs=None,
attention_key_valid=None,
routed_experts=None,
routed=None,
media_mask=None,
admitted=None,
coordinates: Coordinates | None = None,
conditioning: Mapping[str, jax.Array] | None = None,
whole_blocks: bool = False,
progress: Progress | None = None,
)

Return the final normalized states and the prediction depths’ input.

V4’s depth reads the raw residual streams from before the collapse head and the final norm; other depths read the final normalized states. The tokens’ layout is their coordinates, the one description of them every layer reads; only __call__ takes flat layout fields and converts them (token_coordinates, ledger 1122).

  • input_embeddings [B, S, D] replace the scaled token embeddings, for a caller that fuses in another encoder’s output. The token ids still feed the per-layer inputs and routing.
  • image_groups and block_groups [B, S] number the tokens’ images and blocks, -1 outside any, which the attention layers that read them see whole (dew.nn.inputs.AttentionMetadata).
  • attention_pairwise_mask is an explicit [B, queries, keys] visibility mask for ordinary attention, and the optional attention_key_positions [B, keys] give key positions for local windows. Both describe the cached reads of this call only. attention_key_runs [B, queries, R, 2] replaces the mask with each query’s key intervals, whose union it reads, R of them, static, its key positions windowed within each run alone (the template’s spans, dew.nn.attention.scaled_dot_product_attention’s key_runs).
  • routed_experts [B, S, layers, top_k] (or one such a routed site, keyed by it, LayerInputs.experts, where a layer routes at several) replays a rollout engine’s routing (vLLM’s routed_experts, SGLang’s meta_info.routed_experts), and routed [B, S] marks the tokens it covers (dew.nn.protocols.Routes).
  • media_mask [B, S] marks the positions a media encoder fills. Engram keeps them out of its n-grams, and a media-biased router selects for them.
  • admitted makes a cached call a serving step’s mixed call (dew.nn.inputs.Admitted).
  • coordinates are the tokens’ documents, validity and positions as the batch’s builder made them (dew.batch.coordinates, with the packer’s max_segments, and right_padded where each row holds its real tokens first); None for one document a row, every token real.
  • conditioning carries the sequence the cross_attention layers read (dew.nn.inputs.CONTEXT) and its rows’ lengths (CONTEXT_LENGTHS). A cached call that carries it keeps its keys and values for the calls after it. A model without cross-attention takes none.
  • whole_blocks states that each of block_groups’ blocks is its document’s whole run of real tokens (AttentionMetadata.whole_blocks).
  • progress is a training step’s (dew.nn.progress), which its DEX layers anneal over; None is the run’s end, outside a training step.
def looped(x, **stack={})

Run the layer stack over x once, or as loop says (looping.run_loop): its block’s later passes under one scan, so a step’s program holds two passes whatever the loop’s count.

def stack(
x,
*,
train: bool,
decode: bool,
per_layer_input,
attention_metadata=None,
span: tuple[int, int] | None = None,
)

Run the layer stack over x as the plain loop, as scanned runs, or as a pipeline over the stages.

span runs layers [first, end) alone, as a Loop over a block runs its prelude, its block and its coda; on one stage, with no banked store.

init draws like runs under the scan and then unstacks them, so the tree is the plain loop’s; under a stage mesh or with decode, init runs the plain loop. With scan_layers or a stage axis, the stack runs under a StackView, which stacks the leaves while the loops run. The loops carry one residual dtype, so the stream enters in the dtype it settles in (residual_dtype). A store built bank by bank (dew.inference.banks) already holds one array per run, and the view leaves it alone. A pipeline runs the layers its stages repeat (split_stages) and the layers before and after them on every stage, and raises LayoutRefused for decode and for a banked store. With residual_in_fp32 the stream enters widened, so every sublayer’s output adds to it in fp32 whichever loop runs.

def stage_view(stages: int) -> StackView

How a pipeline over stages stacks the run of layers they repeat (split_stages): each stage’s runs as its scan takes them, from the store’s rows (scan_keys).

def stage_sites(store: Variables, stages: int) -> tuple[DecoderBank, ...]

Where the run a pipeline over stages stacks is stored in store, the variables a stage-local state is made from (stage_view): none for a looped stack, which no pipeline runs.

def banked_collections() -> tuple[str, ...]

Return the collections whose layer subtrees the store holds as banks.

A store built per layer holds layers_0, and a store built bank by bank holds each multi-layer run under the run’s name, such as layers_0_15. A run of one layer looks the same either way. A collection that holds a run’s bank beside its layers, with different leaves in each (a host layout’s frozen leaves), is not banked, because its rows still stack (StackView.stack). A bank and a row that hold the same leaf raise ValueError, and so do some banks without the others when there are no rows.

def residual_dtype(
x,
*,
train: bool,
decode: bool,
per_layer_input,
attention_metadata=None,
) -> jnp.dtype

Return the dtype the residual stream settles in: x’s promoted with the first layer’s output.

A scan carries one dtype through, and each pipeline stage takes the dtype the stage before returns. Under a bf16 policy the dense feed-forward returns fp32, so the loops use fp32 from the start. The first layer runs abstractly in its own scope (a placeholder ‘params’ key, which flax draws any other stream from), with the collections the call opened open and no other, so it takes the call’s own path: opened beyond them, an attention sowing its QK-Clip maxima decided a kernel for a mask the call never builds (a cache made under a chosen template refused, row 4101). A banked store answers from its first bank’s first row without fetching it.

def first_layer_shapes() -> dict

Return the stack’s first layer’s variables as shape/dtype structs.

These are layer 0’s variables, a bank’s first row with the layer axis dropped, or layer 0’s variables completed by the bank’s (banked_collections), or a stage-local store’s first stage’s first row of the run that holds layer 0 (StackView.stage_form). Nothing is read, fetched or sliced.

def stage_split(stages: int) -> StageSplit

How the stack runs over stages pipeline stages (split_stages): the layers before and after the run its stages repeat run on every stage.

def per_layer_inputs(tokens, inputs_embeds)

Return every layer’s input signal [B, S, L, P] (Gemma 3n/4 PLE).

The token-identity part is the packed table’s row for each token, scaled like the main embedding. The context part is the input embeddings projected down, scaled and normed. Each layer’s gate multiplies in their sum divided by sqrt(2). A model without per_layer_input_dim raises ValueError.

def output_table() -> OutputTable | None

Return the head as the matrix logits_from_hidden contracts, read from the bound variables (_head_table). None where no matrix alone is the head: past a head_transform or a continuum level, or where lm_head’s or project_out’s parameters are not its kernel alone: dew.lora’s factors beside it, which its interceptor adds, or no kernel, a frozen FP8 layer’s.

def vocabulary_rows() -> jax.Array

Each vocabulary token’s [features] row of the head (OutputTable.rows) where no exact table is: past a head_transform the matrix’s, with an adapter on the head as it updates them (applied_kernel), and past an embedding’s own width in the stream’s, through project_out.

def init_cache(batch_size: int)

Allocate a zeroed decode cache for batch_size sequences.

cache = model.apply(params, batch_size, method=CausalTransformer.init_cache,
mutable=['cache'])[1]['cache']

It runs the forward pass in decode mode on a single dummy token, and that token’s keys are never written: the first decode-mode call allocates the cache, and the calls after it write to it.

def with_kv_cache(layout: KVCache) -> Self

The same decoder over the cache layout layout (Layered).

def with_cache_capacity(capacity: int) -> Self

max_seq_len is the cache size its layers read (open_kv_cache); a learned position table it sized keeps its rows.

def with_passes(
steps: int | None = None,
layers: tuple[int, int] | None = None,
*,
backprop_steps: int | None = None,
remat: bool | None = None,
) -> Self | None

This model with its loop run steps times (None keeps its count), over the block layers where given (None keeps its block). backprop_steps sets how many final passes train through, and remat checkpoints each pass; None keeps the loop’s own value for either.

A model with no loop gains one over layers, with no norm between its passes, so one pass computes the model as it was; None when it has no loop and the two do not both say what to add.

def projected(params, *, updated: bool)

ProjectsParameters: before a training step’s update, every AltUp layer’s prediction coefficients clamped to altup.coef_clip, as the reference clamps its stored weight before a training forward (clamped_predictions); the updated parameters as they are.

def with_cross_attention(cross: CrossBlocks | None) -> Self

This decoder with cross as its cross-attention layers (CrossAttending), None for none.

def with_dex(extension: DifferentialExtension | None) -> Self

This decoder with extension’s differential on its chosen heads (dew.nn.dex), None for none.

def with_scanned_layers() -> Self

This decoder running its like and periodic runs under the scan (scan_layers).

def reads_progress() -> bool

Whether its computation changes over a run (dew.nn.progress.ReadsProgress): its DEX layers’ lambda anneals over the run’s updates.

def dex_exported(variables: Mapping[str, Mapping]) -> dict[str, Mapping]

variables as an export writes them, each DEX layer’s differential folded into its output projection at the operands its layers multiply in (dew.nn.dex.folded); ValueError names a layer whose heads’ outputs pass through more before it, which does not fold.

def dex_sites() -> tuple[tuple[str, ...], frozenset[int]]

Its parameters at the root of params, and its softmax attention layers (Differential).

def with_remat(policy: RematPolicy | None) -> Self

Start at policy, including policies outside DECODER_REMAT.

recompute_more stays at a policy off that ladder, so the trainer cannot climb from it; None keeps every layer’s residuals.

def recompute_record() -> JSON
def recompute_more() -> Self | None

One rung up DECODER_REMAT, past any rung that keeps nothing more of the names these layers mark than the rung above it (MixerBase.marks), whose step would compile the same; None at the top, off the ladder, or where the top keeps what this rung keeps, so no rung above holds less.

def recompute_offloaded() -> Self | None

At the ladder’s top (‘full’), the decoder that keeps what ‘minimal’ saves in pinned host memory (‘minimal_offloaded’): as many device bytes as ‘full’ (tests/test_remat_rungs.py’s _saved), the recompute of ‘minimal’ and its residuals over the host link (row 1932); else None.

def recompute_half() -> Self | None

The decoder one rung up where that rung recomputes only the residuals it names (RematPolicy.recompute, a delta rule’s memory), else None.

def marked_residuals() -> frozenset[str]

The RESIDUALS this decoder’s layers can mark: each layer’s mixer kind’s (MixerBase.marks, every name where a kind does not say) and, where a layer, its gated cross-attention, a prediction depth or the drafter has a feed-forward, its (MLP_RESIDUALS, the dense MLP’s and the experts’ alike).

def restore_recompute(record: JSON) -> Self

CausalTransformer.inference_projection_groups

Section titled “CausalTransformer.inference_projection_groups”
def inference_projection_groups(
variables: Mapping[str, Mapping],
) -> tuple[ProjectionGroup, ...]

Its layers’, depths’ and drafter’s groups, bound without tracing a forward, causal or not: a bidirectional model’s masked denoising reads its projections packed as a decoder’s steps do. The groups read which parameters the layers hold, never their values, so the layers are bound over the variables’ shapes: flax’s bound tree is cyclic, and bound over the weights themselves it would keep them alive after the caller drops them until a full collection.

def block_chunks() -> tuple[int | None, ...]

The attention chunk of each trunk layer that reads a block whole (MixerBase.reads_blocks), None where a layer has none (BlockReading). Each layer’s mixer is _layer_specs’ own, the kind’s or the model’s layer mixer, so a prediction depth’s or the drafter’s kind, which the trunk never runs, reads none here.

def prediction_kind(types: Sequence[str]) -> str

A prediction depth’s mixer kind: mtp_layer_type, else full attention, else the first layer’s.

def fixed_routers() -> Mapping[str, Callable[[tuple[str, ...]], bool]]

Its routers whose weights are fixed, and their parameters (layer_names.fixed_router_owners).

def mixed_admission_refusal() -> str | None

A cache rebuilt per request, a reader beyond the token, a context read beside the tokens, or a layer whose mixer, over the context it is built with, would not run the mixed call as the separate calls would (MixerBase.mixed_refusal).

def generation_refusal() -> str | None

Why a generation loop cannot continue this decoder’s rows, or None (dew.nn.protocols.Generating): its sparse layers’ mixture, or a mixer it names (declared_mixers: a layer kind’s, a prediction depth’s or the drafter’s), that refuses it (Mixture.generation_refusal, MixerBase.generation_refusal).

def split_prefill_refusal() -> str | None

A cache rebuilt per request from its whole prompt, or a loop’s shared cache that a later pass rewrites, whose earlier tokens hold the last pass’s keys: a prompt’s later piece would read those where its whole prefill reads each pass’s own. One pass, or a block whose layers keep no cache (MixerBase.keeps_cache), rewrites nothing.