dew.nn.kv_cache
How a decode cache stores its keys and values: dense or paged, full precision or quantized.
open_kv_cache keeps the cursor and the compact slot positions. A KVCache
value names the storage layout behind them, and KVStore reads and writes
that storage inside one attention module.
Dense storage is one [rows, capacity, kv_heads, head_dim] block per row.
Paged storage is one pool [kv_heads, pages, page_size, head_dim] shared by
every row, which is the layout the Pallas TPU paged attention kernel reads.
A [rows, capacity // page_size] page table maps slot s to page
table[row, s // page_size] at offset s % page_size. With pages, a server
can assign pages on demand, share the pages of a common prefix, and hold more
than pages * page_size / capacity rows while they are short. A pool split
into groups gives each equal group of rows its own part of the pool,
indexed from the part’s start. A server whose mesh splits rows n ways then
reads and writes only the pages each device holds, and does not gather the
whole pool.
A quantized cache stores int8 or float8 (e4m3) values with one float32 scale
per token and head, and dequantizes them on read. The scale is the absmax
over the head’s features divided by the format’s largest value. int8 keys
are stored rotated by the orthonormal Hadamard matrix over head_dim, the way
QuaRot (arXiv:2404.00456) rotates its cache, so a few outlier channels
(Qwen3’s keys) do not set the whole per-token scale. The query gets the same
rotation (Append.query), and every logit stays the same because
H Hᵀ = I. Values and float8 keys are not rotated, because e4m3 rounds each
element relative to itself. On Qwen3-0.6B over wikitext-2, compared with a
bf16 cache, perplexity changed by +0.66 for unrotated int8 and +0.008 for
rotated int8, and by -0.03 for unrotated float8 and +1.03 for rotated float8
(docs/concepts/inference.md).
| Name | Summary |
|---|---|
Append | Stores a call’s keys and values, then reads the cache; open_kv_cache returns one. |
Banked | One layer’s key and value storage leaf (POOLED) as it lies in its scanned run’s bank: the bank [layers, ...] and the layer, which KVStore reads at the layer and writes in place, a step’s tokens at their slots alone. |
KVCache | The storage layout of a decode cache. |
KVStore | Reads and writes one attention module’s key and value storage, opened for rows rows. |
Append
Section titled “Append”class Append(store: KVStore, positions: jax.Array, allocated: bool)Stores a call’s keys and values, then reads the cache; open_kv_cache returns one.
The allocation-only call (when the cache did not exist yet) stores
nothing. The keys it reads back are in the store’s rotation, so the query
that attends to them must go through query first.
Append.query
Section titled “Append.query”def query(query: jax.Array) -> jax.ArrayReturn query [..., head_dim] in the basis that the stored keys are in.
Append.places
Section titled “Append.places”def places(places: jax.Array) -> jax.ArrayStore the call’s token places beside its keys and return every slot’s, [rows, capacity].
places are [rows, tokens] or one vector the rows share. A chunk
and ALiBi read a cached key at the place the call that wrote it
counted its token at (dew.nn.inputs.counted_places). A caller may
place a row’s tokens anywhere, so a slot is no place of its own, and a
paged pool keeps the places in its pages, which a shared prefix’s
pages then carry.
Append.groups
Section titled “Append.groups”def groups(name: str, groups: jax.Array | None) -> jax.ArrayStore the call’s tokens’ groups of field name [rows, tokens] beside their keys, -1 where it
carries none, and return every slot’s, [rows, capacity]: a later call’s token reads its block’s
earlier keys by them, and a layer sharing these keys reads them too.
Banked
Section titled “Banked”class Banked(struct.PyTreeNode)One layer’s key and value storage leaf (POOLED) as it lies in its scanned run’s bank: the bank
[layers, ...] and the layer, which KVStore reads at the layer and writes in place, a step’s tokens
at their slots alone. A layer handed its slice instead copies it out of the bank, and the bank takes
the whole slice back after the layer writes it: at Qwen3-30B-A3B’s attention over 64 rows of 512 slots
on an H100, 45 us copying a layer’s keys and values out and 47 writing them back, beside the
attention’s own 33 (OM-1342-cache-base-2474a6e4c).
Banked.read
Section titled “Banked.read”def read() -> jax.ArrayThe layer’s leaf, read where the bank holds it: the slice a reader fuses.
KVCache
Section titled “KVCache”class KVCache( quantized: KVDtype | None = None, page_size: int | None = None, pages: int | None = None, groups: int = 1,)The storage layout of a decode cache.
quantized stores keys and values in that format with per-token, per-head
scales. int8 keys are Hadamard-rotated, which needs a power-of-two
head_dim. page_size pages the cache, and capacity must be a multiple of
it. pages is the size of the pool. None gives every row enough pages for
its whole capacity, and a smaller pool is for a server to assign
(dew.inference.serving.Server). groups splits a paged pool among equal
groups of rows, so the row count and pages must both divide by it.
KVCache.storage
Section titled “KVCache.storage”def storage(dtype: jnp.dtype) -> jnp.dtypeReturn the dtype that the cache stores values of dtype in.
KVCache.key_rotation
Section titled “KVCache.key_rotation”def key_rotation(head_dim: int) -> jax.Array | NoneReturn the rotation that stored keys are in and a query must take, or None for unrotated keys.
KVStore
Section titled “KVStore”class KVStore( module: nn.Module, layout: KVCache, rows: int, kv_heads: int, head_dim: int, dtype: jnp.dtype, keys_as_values: bool = False,)Reads and writes one attention module’s key and value storage, opened for rows rows.
write puts keys [rows, tokens, kv_heads, head_dim] and values as wide
as value_dim at
compact slot positions [rows, tokens] and drops the ones at -1. read
returns every slot of every row, [rows, capacity, kv_heads, head_dim],
dequantized and with the keys in the stored rotation. A slot that no token
has reached holds whatever its page last held, and the caller’s validity
mask excludes it. A store whose values are its keys (keys_as_values, a
head that serves as both) keeps them once where its keys are stored as
given, with no rotation, and reads them back as its values; a rotated
layout’s keys are not its values’ layout, so it keeps both.
payloads: tuple[tuple[str, str], ...]-
The leaves the store keeps, each beside its scale’s: the keys’, and the values’ unless the keys serve as them, stored as given (
keys_as_values).
KVStore.open
Section titled “KVStore.open”def open( module: nn.Module, layout: KVCache, rows: int, capacity: int, kv_heads: int, head_dim: int, dtype: jnp.dtype, *, value_dim: int | None = None, keys_as_values: bool = False, places: bool = False, groups: tuple[str, ...] = (),) -> KVStoreDeclare the module’s cache variables, allocating them on first use, and return the store.
Values are value_dim wide, None for the keys’ head_dim, or are the keys themselves where
keys_as_values and the layout stores keys as given (no rotation), which allocates no storage
of their own. places
also keeps each token’s place beside its key (Append.places), and
groups each token’s group of the fields it names (GROUPS,
Append.groups), -1 where none has been written. A paged
layout raises ValueError when capacity is not a multiple of
page_size, or when rows does not split into the pool’s groups.
KVStore.write
Section titled “KVStore.write”def write(key: jax.Array, value: jax.Array, positions: jax.Array) -> NoneStore key and value at positions, value once where it is the keys (payloads).
KVStore.plain
Section titled “KVStore.plain”def plain(keys: jax.Array) -> jax.ArrayReturn keys read back in the store’s rotation [..., head_dim] in their own basis: for keys a
projection expands before any query meets them (a latent attention’s latents), or that serve as
values too (DeepSeek V4’s shared head), where no query is rotated into their basis.
KVStore.read
Section titled “KVStore.read”def read() -> tuple[jax.Array, jax.Array]KVStore.write_places
Section titled “KVStore.write_places”def write_places(places: jax.Array, positions: jax.Array) -> NoneKeep the tokens’ places [rows, tokens] at their slots positions, where their keys go.
KVStore.read_places
Section titled “KVStore.read_places”def read_places() -> jax.ArrayEvery slot’s kept place, [rows, capacity].
KVStore.write_groups
Section titled “KVStore.write_groups”def write_groups(name: str, groups: jax.Array, positions: jax.Array) -> NoneKeep the tokens’ groups of field name [rows, tokens] at their slots positions, where their
keys go (GROUPS).
KVStore.read_groups
Section titled “KVStore.read_groups”def read_groups(name: str) -> jax.ArrayEvery slot’s kept group of field name, [rows, capacity].
KVStore.write_tokens
Section titled “KVStore.write_tokens”def write_tokens( key: jax.Array, value: jax.Array, rows: jax.Array, positions: jax.Array,) -> NoneWrite keys and values [tokens, kv_heads, head_dim], each at its own row and slot.
This is the write of a serving step’s mixed call
(dew.nn.inputs.Admitted). A token whose row is past the cache’s rows,
or whose slot is -1, is dropped. The cache must be full precision and
unrotated, and a paged pool must have one group, as the mixed call
requires.
KVStore.read_rows
Section titled “KVStore.read_rows”def read_rows(rows: jax.Array) -> tuple[jax.Array, jax.Array]Return what read returns for only rows, [len(rows), capacity, kv_heads, head_dim].
It does not dequantize, so it matches read only on a full-precision
cache, which the mixed call that uses it requires.
KVStore.attends_rows
Section titled “KVStore.attends_rows”def attends_rows(query: jax.Array, softcap: float | None) -> boolWhether attend_rows reads this cache for query: its rows’ keys of the query’s 16-bit dtype,
unquantized (the template’s decode reads no codes yet, ledger 3624), a paged pool’s pages each
whole tiles or tiles of whole pages (TEMPLATE_PAGED_SUPPORTS), on a GPU its Mosaic mainloop is
qualified on.
KVStore.attend_rows
Section titled “KVStore.attend_rows”def attend_rows( query: jax.Array, spans, sources: jax.Array, softcap: float | None,) -> jax.Arrayquery [B, T, H, D] through the attention template’s decode over this cache’s rows
(attends_rows):
batch row b over row sources[b]’s keys under spans (attention.KeySpans), read where they
lie: a dense cache’s rows by sources, a paged pool’s by those rows’ tables. One call over a serving
step’s work list (ledger 3623).
KVStore.case
Section titled “KVStore.case”def case( query: jax.Array, *, value_dim: int, softcap: float | None, arithmetic: bool, named: str, windowed: bool = False, sinks: bool = False,) -> PagedCaseThe PagedCase of a decode step’s query [rows, heads, head_dim] over this pool, its values
value_dim wide, under the layer’s softcap, its reference-only arithmetic
(dew.nn.attention.reference_only) and its AttentionImpl, named; a windowed step reads
each row from its window’s first key; a layer with sinks has attention sinks.
KVStore.decode
Section titled “KVStore.decode”def decode( query: jax.Array, lengths: jax.Array, softcap: float | None, starts: jax.Array | None = None, sinks: jax.Array | None = None,) -> jax.ArrayAttend one query per row [rows, heads, head_dim] to the first lengths slots of that row,
from starts [rows] where given (a window’s first slot, which the GPU’s kernel reads), with
sinks [heads] where the layer has them (the GPU’s kernel applies them).
Decode runs the paged kernel the selector takes (PAGED: the device’s, or the attention template’s
decode where the GPU’s refuses the pool), which raises KernelRefused where every paged kernel’s
requirements turn the step down. The TPU kernel does not scale
the logits, so the query is multiplied by 1/sqrt(head_dim) first, the
same scale every other attention path applies. A Mosaic kernel cannot
be partitioned automatically, and neither can the GPU’s Pallas kernel,
so on a mesh each runs per shard (manual_map): each holds its rows’
queries, lengths and page tables, and the key heads of the pool its
query heads read, with every page, as the tables index the whole pool.