dew.nn.engram
Engram: n-gram hash lookups written into the residual streams.
DeepSeek-V4.1 (arXiv 2609.19969, section 2.4.2) adds Engram (Cheng et al.
2026) at a few layers; the reference is the release’s engram.py and
model.py (DEEPSEEK_V41_REVISION in tools/deepseek_v41_reference.py). Per
position:
- Every token id maps through a compressed vocabulary of normalized ids
(
compressed_token_map, engram.py:17-55). - The position and the
max_ngram_size - 1compressed ids before it make one n-gram of each size from 2 up; look-back stops at the sequence start and at a dead token (an image span), and missing slots hold the pad id’s compressed id (engram.py:155-170). - Each layer multiplies the look-back ids by its own odd multipliers and
XORs them, so after
isteps the value hashes the(i + 1)-gram; each (n-gram size, head) reduces it modulo its prime and offsets it into its bucket range (engram.py:172-180). - The layer projects the looked-up rows into one key per residual stream and a shared value, gated into each stream by a sigmoid of the sign-preserving square root of the stream’s normalized dot product with its key (model.py:328-365).
The reference hashes in 64-bit integers; here 8-bit limbs in uint32 stay exact without x64, since every compressed id stays under 224 and every prime under 228.
Qwen4-Exp’s n-gram per-layer embedding (Qwen4ExpTextNGramEmbedding and
Qwen4ExpTextPLELayer, modeling_qwen4_exp.py:971-1190 in transformers
5.16.1) hashes the same way over raw token ids: its multipliers come from a
SplitMix64 stream (Engram.multiplier_seed), its look-back stops at an
end-of-sequence id, which fills the slots it stops (stop_token_id,
pad_token_id), and each table’s rows round up to a multiple (row_multiple).
Its layer (PLELayer) gates one shared value into every stream by the same
sign-preserving root, then adds a dilated causal conv of the gated values.
| Name | Summary |
|---|---|
Engram | The engram layers and their hash layout, by the release’s config names. |
PLELayer | Qwen4-Exp’s n-gram per-layer embedding, added to the residual streams (Qwen4ExpTextPLELayer, modeling_qwen4_exp.py:1117-1190), which is DeepSeek’s demo Engram as MaxText computes it (maxtext/layers/engram.py:496-695), its projections biased there (use_bias). |
compressed_token_map | Every token id onto the compressed vocabulary, and its size (engram.py:17-55): a token’s text through NFKC, NFD, accent stripping, lowercasing, whitespace runs to one space and a strip that leaves a lone space standing; a partial UTF-8 token is keyed by its raw form. |
Engram
Section titled “Engram”class Engram( layer_ids: tuple[int, ...], num_embeddings: tuple[int, ...], max_ngram_size: int, vocab_size: int, n_heads: int, head_dim: int, compressed_vocab_size: int, pad_token_id: int = 2, multiplier_seed: int | None = None, stop_token_id: int | None = None, row_multiple: int = 1, ple_conv_kernel_size: int | None = None, ple_bias: bool = False, table_parts: int = 1, media_stops: bool = True,)The engram layers and their hash layout, by the release’s config names.
layer_ids are the decoder layers that add a lookup before their
attention; num_embeddings their table rows (none stated: the primes’
layout), vocab_size the bucket
count each prime search starts above, compressed_vocab_size the size of
the tokenizer’s compressed vocabulary every multiplier derives from, and
pad_token_id the vocabulary id whose compressed id fills missing
look-back (engram_pad_token_id).
multiplier_seed: int | None-
Qwen4-Exp’s
seed: each table’s multipliers drawn from SplitMix64 atseed + 10007 * index, the table’s index among the lookups (_build_layer_multipliers, modeling_qwen4_exp.py:986-996); None draws them as V4.1 does (multipliers). stop_token_id: int | None-
A token id a position’s look-back stops at, filling that slot and every one past it with the pad id: Qwen4-Exp’s end of sequence (
_shift_right_ignore_eos, :1053-1067); None for none. row_multiple: int-
Each table’s rows, the primes’ sum rounded up to this multiple (make_ngram_vocab_size_divisible_by).
ple_conv_kernel_size: int | None-
Qwen4-Exp’s per-layer embedding: each lookup a
PLELayer, its conv of this many tapsmax_ngram_sizeapart; None for V4.1’sEngramLayer. ple_bias: bool-
Whether a
PLELayer’s key and value projections add biases: DeepSeek’s demo Engram’s and MaxText’s (maxtext/layers/engram.py), whose layer the PLE layer computes; Qwen4-Exp’s add none. table_parts: int-
The equal row shards each table is stored in (Qwen4-Exp’s
split_ngram_parts), a PLE layer’s table one leaf of them stacked (PLELayer), each written back as stored. media_stops: bool-
Whether a media position is dead, its look-back stopped there (V4.1, engram.py:164-166); Qwen4-Exp hashes an image’s placeholder ids as it does any token’s (
ple_input_ids, modeling_qwen4_exp.py:1361). primes: tuple[tuple[tuple[int, ...], ...], ...]-
[layer][n-gram size][head]bucket moduli: the next unused prime abovevocab_size - 1, drawn in order (engram.py:9-14, :107-118). Each search restarts there and skips the primes drawn before it, so the draws are the successive primes fromvocab_sizeon. multipliers: np.ndarray-
[layers, max_ngram_size]odd multipliers, bounded so a product fits int64: from a per-layer RNG seeded 10007 * layer (engram.py:58-75), or undermultiplier_seedfrom SplitMix64 atseed + 10007 * index(modeling_qwen4_exp.py:986-996).
Engram.hash_ids
Section titled “Engram.hash_ids”def hash_ids(compressed, blocked, pad)Every engram layer’s bucket ids, [B, S, layers, columns] int32,
a column per (n-gram size, head).
compressed [B, S, max_ngram_size] holds each position’s compressed
id and its look-back, nearest first, with blocked marking the slots
past the sequence start or a dead token; those hash as pad, the pad
token’s compressed id.
PLELayer
Section titled “PLELayer”class PLELayer( parts: int = 1, use_bias: bool = False, dtype: Dtype | None = None, precision: PrecisionLike = None,)Qwen4-Exp’s n-gram per-layer embedding, added to the residual streams
(Qwen4ExpTextPLELayer, modeling_qwen4_exp.py:1117-1190), which is DeepSeek’s demo Engram as
MaxText computes it (maxtext/layers/engram.py:496-695), its projections biased there (use_bias).
The looked-up rows of shards (ShardedTable) project to one key per
stream (key_proj) and one shared value (value_proj). Each stream’s
normed activations meet its normed key, and the sign-preserving root of
their scaled product gates the value into it; the gated values, normed
again, run through a causal depthwise conv of kernel_size taps
dilation apart and a SiLU, which the gated values themselves join. The
three norms take each stream’s own statistics under one (1 + w) weight
(RMSNorm(groups=hc_mult)). The call’s padded slots (call.metadata) add
nothing and the conv steps over them and across no document’s start
(modeling_qwen4_exp.py:1176-1190); under call.decode the conv’s last
(kernel_size - 1) * dilation inputs wait in the cache collection’s
conv_state, which the first decode call only allocates.
compressed_token_map
Section titled “compressed_token_map”def compressed_token_map(tokenizer) -> tuple[np.ndarray, int]Every token id onto the compressed vocabulary, and its size
(engram.py:17-55): a token’s text through NFKC, NFD, accent stripping,
lowercasing, whitespace runs to one space and a strip that leaves a lone
space standing; a partial UTF-8 token is keyed by its raw form. Ids
number the keys in first-seen order. tokenizer is a transformers fast
tokenizer; the raw Rust tokenizer decodes, as training did.