Skip to content

dew.nn.engram

Engram: n-gram hash lookups written into the residual streams.

DeepSeek-V4.1 (arXiv 2609.19969, section 2.4.2) adds Engram (Cheng et al. 2026) at a few layers; the reference is the release’s engram.py and model.py (DEEPSEEK_V41_REVISION in tools/deepseek_v41_reference.py). Per position:

  1. Every token id maps through a compressed vocabulary of normalized ids (compressed_token_map, engram.py:17-55).
  2. The position and the max_ngram_size - 1 compressed ids before it make one n-gram of each size from 2 up; look-back stops at the sequence start and at a dead token (an image span), and missing slots hold the pad id’s compressed id (engram.py:155-170).
  3. Each layer multiplies the look-back ids by its own odd multipliers and XORs them, so after i steps the value hashes the (i + 1)-gram; each (n-gram size, head) reduces it modulo its prime and offsets it into its bucket range (engram.py:172-180).
  4. The layer projects the looked-up rows into one key per residual stream and a shared value, gated into each stream by a sigmoid of the sign-preserving square root of the stream’s normalized dot product with its key (model.py:328-365).

The reference hashes in 64-bit integers; here 8-bit limbs in uint32 stay exact without x64, since every compressed id stays under 224 and every prime under 228.

Qwen4-Exp’s n-gram per-layer embedding (Qwen4ExpTextNGramEmbedding and Qwen4ExpTextPLELayer, modeling_qwen4_exp.py:971-1190 in transformers 5.16.1) hashes the same way over raw token ids: its multipliers come from a SplitMix64 stream (Engram.multiplier_seed), its look-back stops at an end-of-sequence id, which fills the slots it stops (stop_token_id, pad_token_id), and each table’s rows round up to a multiple (row_multiple). Its layer (PLELayer) gates one shared value into every stream by the same sign-preserving root, then adds a dilated causal conv of the gated values.

NameSummary
EngramThe engram layers and their hash layout, by the release’s config names.
PLELayerQwen4-Exp’s n-gram per-layer embedding, added to the residual streams (Qwen4ExpTextPLELayer, modeling_qwen4_exp.py:1117-1190), which is DeepSeek’s demo Engram as MaxText computes it (maxtext/layers/engram.py:496-695), its projections biased there (use_bias).
compressed_token_mapEvery token id onto the compressed vocabulary, and its size (engram.py:17-55): a token’s text through NFKC, NFD, accent stripping, lowercasing, whitespace runs to one space and a strip that leaves a lone space standing; a partial UTF-8 token is keyed by its raw form.

dataclass source

class Engram(
layer_ids: tuple[int, ...],
num_embeddings: tuple[int, ...],
max_ngram_size: int,
vocab_size: int,
n_heads: int,
head_dim: int,
compressed_vocab_size: int,
pad_token_id: int = 2,
multiplier_seed: int | None = None,
stop_token_id: int | None = None,
row_multiple: int = 1,
ple_conv_kernel_size: int | None = None,
ple_bias: bool = False,
table_parts: int = 1,
media_stops: bool = True,
)

The engram layers and their hash layout, by the release’s config names.

layer_ids are the decoder layers that add a lookup before their attention; num_embeddings their table rows (none stated: the primes’ layout), vocab_size the bucket count each prime search starts above, compressed_vocab_size the size of the tokenizer’s compressed vocabulary every multiplier derives from, and pad_token_id the vocabulary id whose compressed id fills missing look-back (engram_pad_token_id).

multiplier_seed: int | None

Qwen4-Exp’s seed: each table’s multipliers drawn from SplitMix64 at seed + 10007 * index, the table’s index among the lookups (_build_layer_multipliers, modeling_qwen4_exp.py:986-996); None draws them as V4.1 does (multipliers).

stop_token_id: int | None

A token id a position’s look-back stops at, filling that slot and every one past it with the pad id: Qwen4-Exp’s end of sequence (_shift_right_ignore_eos, :1053-1067); None for none.

row_multiple: int

Each table’s rows, the primes’ sum rounded up to this multiple (make_ngram_vocab_size_divisible_by).

ple_conv_kernel_size: int | None

Qwen4-Exp’s per-layer embedding: each lookup a PLELayer, its conv of this many taps max_ngram_size apart; None for V4.1’s EngramLayer.

ple_bias: bool

Whether a PLELayer’s key and value projections add biases: DeepSeek’s demo Engram’s and MaxText’s (maxtext/layers/engram.py), whose layer the PLE layer computes; Qwen4-Exp’s add none.

table_parts: int

The equal row shards each table is stored in (Qwen4-Exp’s split_ngram_parts), a PLE layer’s table one leaf of them stacked (PLELayer), each written back as stored.

media_stops: bool

Whether a media position is dead, its look-back stopped there (V4.1, engram.py:164-166); Qwen4-Exp hashes an image’s placeholder ids as it does any token’s (ple_input_ids, modeling_qwen4_exp.py:1361).

primes: tuple[tuple[tuple[int, ...], ...], ...]

[layer][n-gram size][head] bucket moduli: the next unused prime above vocab_size - 1, drawn in order (engram.py:9-14, :107-118). Each search restarts there and skips the primes drawn before it, so the draws are the successive primes from vocab_size on.

multipliers: np.ndarray

[layers, max_ngram_size] odd multipliers, bounded so a product fits int64: from a per-layer RNG seeded 10007 * layer (engram.py:58-75), or under multiplier_seed from SplitMix64 at seed + 10007 * index (modeling_qwen4_exp.py:986-996).

def hash_ids(compressed, blocked, pad)

Every engram layer’s bucket ids, [B, S, layers, columns] int32, a column per (n-gram size, head).

compressed [B, S, max_ngram_size] holds each position’s compressed id and its look-back, nearest first, with blocked marking the slots past the sequence start or a dead token; those hash as pad, the pad token’s compressed id.

Flax module source

class PLELayer(
parts: int = 1,
use_bias: bool = False,
dtype: Dtype | None = None,
precision: PrecisionLike = None,
)

Qwen4-Exp’s n-gram per-layer embedding, added to the residual streams (Qwen4ExpTextPLELayer, modeling_qwen4_exp.py:1117-1190), which is DeepSeek’s demo Engram as MaxText computes it (maxtext/layers/engram.py:496-695), its projections biased there (use_bias).

The looked-up rows of shards (ShardedTable) project to one key per stream (key_proj) and one shared value (value_proj). Each stream’s normed activations meet its normed key, and the sign-preserving root of their scaled product gates the value into it; the gated values, normed again, run through a causal depthwise conv of kernel_size taps dilation apart and a SiLU, which the gated values themselves join. The three norms take each stream’s own statistics under one (1 + w) weight (RMSNorm(groups=hc_mult)). The call’s padded slots (call.metadata) add nothing and the conv steps over them and across no document’s start (modeling_qwen4_exp.py:1176-1190); under call.decode the conv’s last (kernel_size - 1) * dilation inputs wait in the cache collection’s conv_state, which the first decode call only allocates.

function source

def compressed_token_map(tokenizer) -> tuple[np.ndarray, int]

Every token id onto the compressed vocabulary, and its size (engram.py:17-55): a token’s text through NFKC, NFD, accent stripping, lowercasing, whitespace runs to one space and a strip that leaves a lone space standing; a partial UTF-8 token is keyed by its raw form. Ids number the keys in first-seen order. tokenizer is a transformers fast tokenizer; the raw Rust tokenizer decodes, as training did.