Skip to content

dew.nn.kv_cache

How a decode cache stores its keys and values: dense or paged, full precision or quantized.

open_kv_cache keeps the cursor and the compact slot positions. A KVCache value names the storage layout behind them, and KVStore reads and writes that storage inside one attention module.

Dense storage is one [rows, capacity, kv_heads, head_dim] block per row. Paged storage is one pool [kv_heads, pages, page_size, head_dim] shared by every row, which is the layout the Pallas TPU paged attention kernel reads. A [rows, capacity // page_size] page table maps slot s to page table[row, s // page_size] at offset s % page_size. With pages, a server can assign pages on demand, share the pages of a common prefix, and hold more than pages * page_size / capacity rows while they are short. A pool split into groups gives each equal group of rows its own part of the pool, indexed from the part’s start. A server whose mesh splits rows n ways then reads and writes only the pages each device holds, and does not gather the whole pool.

A quantized cache stores int8 or float8 (e4m3) values with one float32 scale per token and head, and dequantizes them on read. The scale is the absmax over the head’s features divided by the format’s largest value. int8 keys are stored rotated by the orthonormal Hadamard matrix over head_dim, the way QuaRot (arXiv:2404.00456) rotates its cache, so a few outlier channels (Qwen3’s keys) do not set the whole per-token scale. The query gets the same rotation (Append.query), and every logit stays the same because H Hᵀ = I. Values and float8 keys are not rotated, because e4m3 rounds each element relative to itself. On Qwen3-0.6B over wikitext-2, compared with a bf16 cache, perplexity changed by +0.66 for unrotated int8 and +0.008 for rotated int8, and by -0.03 for unrotated float8 and +1.03 for rotated float8 (docs/concepts/inference.md).

NameSummary
AppendStores a call’s keys and values, then reads the cache; open_kv_cache returns one.
BankedOne layer’s key and value storage leaf (POOLED) as it lies in its scanned run’s bank: the bank [layers, ...] and the layer, which KVStore reads at the layer and writes in place, a step’s tokens at their slots alone.
KVCacheThe storage layout of a decode cache.
KVStoreReads and writes one attention module’s key and value storage, opened for rows rows.

dataclass source

class Append(store: KVStore, positions: jax.Array, allocated: bool)

Stores a call’s keys and values, then reads the cache; open_kv_cache returns one.

The allocation-only call (when the cache did not exist yet) stores nothing. The keys it reads back are in the store’s rotation, so the query that attends to them must go through query first.

def query(query: jax.Array) -> jax.Array

Return query [..., head_dim] in the basis that the stored keys are in.

def places(places: jax.Array) -> jax.Array

Store the call’s token places beside its keys and return every slot’s, [rows, capacity].

places are [rows, tokens] or one vector the rows share. A chunk and ALiBi read a cached key at the place the call that wrote it counted its token at (dew.nn.inputs.counted_places). A caller may place a row’s tokens anywhere, so a slot is no place of its own, and a paged pool keeps the places in its pages, which a shared prefix’s pages then carry.

def groups(name: str, groups: jax.Array | None) -> jax.Array

Store the call’s tokens’ groups of field name [rows, tokens] beside their keys, -1 where it carries none, and return every slot’s, [rows, capacity]: a later call’s token reads its block’s earlier keys by them, and a layer sharing these keys reads them too.

class source

class Banked(struct.PyTreeNode)

One layer’s key and value storage leaf (POOLED) as it lies in its scanned run’s bank: the bank [layers, ...] and the layer, which KVStore reads at the layer and writes in place, a step’s tokens at their slots alone. A layer handed its slice instead copies it out of the bank, and the bank takes the whole slice back after the layer writes it: at Qwen3-30B-A3B’s attention over 64 rows of 512 slots on an H100, 45 us copying a layer’s keys and values out and 47 writing them back, beside the attention’s own 33 (OM-1342-cache-base-2474a6e4c).

def read() -> jax.Array

The layer’s leaf, read where the bank holds it: the slice a reader fuses.

dataclass source

class KVCache(
quantized: KVDtype | None = None,
page_size: int | None = None,
pages: int | None = None,
groups: int = 1,
)

The storage layout of a decode cache.

quantized stores keys and values in that format with per-token, per-head scales. int8 keys are Hadamard-rotated, which needs a power-of-two head_dim. page_size pages the cache, and capacity must be a multiple of it. pages is the size of the pool. None gives every row enough pages for its whole capacity, and a smaller pool is for a server to assign (dew.inference.serving.Server). groups splits a paged pool among equal groups of rows, so the row count and pages must both divide by it.

def storage(dtype: jnp.dtype) -> jnp.dtype

Return the dtype that the cache stores values of dtype in.

def key_rotation(head_dim: int) -> jax.Array | None

Return the rotation that stored keys are in and a query must take, or None for unrotated keys.

dataclass source

class KVStore(
module: nn.Module,
layout: KVCache,
rows: int,
kv_heads: int,
head_dim: int,
dtype: jnp.dtype,
keys_as_values: bool = False,
)

Reads and writes one attention module’s key and value storage, opened for rows rows.

write puts keys [rows, tokens, kv_heads, head_dim] and values as wide as value_dim at compact slot positions [rows, tokens] and drops the ones at -1. read returns every slot of every row, [rows, capacity, kv_heads, head_dim], dequantized and with the keys in the stored rotation. A slot that no token has reached holds whatever its page last held, and the caller’s validity mask excludes it. A store whose values are its keys (keys_as_values, a head that serves as both) keeps them once where its keys are stored as given, with no rotation, and reads them back as its values; a rotated layout’s keys are not its values’ layout, so it keeps both.

payloads: tuple[tuple[str, str], ...]

The leaves the store keeps, each beside its scale’s: the keys’, and the values’ unless the keys serve as them, stored as given (keys_as_values).

def open(
module: nn.Module,
layout: KVCache,
rows: int,
capacity: int,
kv_heads: int,
head_dim: int,
dtype: jnp.dtype,
*,
value_dim: int | None = None,
keys_as_values: bool = False,
places: bool = False,
groups: tuple[str, ...] = (),
) -> KVStore

Declare the module’s cache variables, allocating them on first use, and return the store.

Values are value_dim wide, None for the keys’ head_dim, or are the keys themselves where keys_as_values and the layout stores keys as given (no rotation), which allocates no storage of their own. places also keeps each token’s place beside its key (Append.places), and groups each token’s group of the fields it names (GROUPS, Append.groups), -1 where none has been written. A paged layout raises ValueError when capacity is not a multiple of page_size, or when rows does not split into the pool’s groups.

def write(key: jax.Array, value: jax.Array, positions: jax.Array) -> None

Store key and value at positions, value once where it is the keys (payloads).

def plain(keys: jax.Array) -> jax.Array

Return keys read back in the store’s rotation [..., head_dim] in their own basis: for keys a projection expands before any query meets them (a latent attention’s latents), or that serve as values too (DeepSeek V4’s shared head), where no query is rotated into their basis.

def read() -> tuple[jax.Array, jax.Array]
def write_places(places: jax.Array, positions: jax.Array) -> None

Keep the tokens’ places [rows, tokens] at their slots positions, where their keys go.

def read_places() -> jax.Array

Every slot’s kept place, [rows, capacity].

def write_groups(name: str, groups: jax.Array, positions: jax.Array) -> None

Keep the tokens’ groups of field name [rows, tokens] at their slots positions, where their keys go (GROUPS).

def read_groups(name: str) -> jax.Array

Every slot’s kept group of field name, [rows, capacity].

def write_tokens(
key: jax.Array,
value: jax.Array,
rows: jax.Array,
positions: jax.Array,
) -> None

Write keys and values [tokens, kv_heads, head_dim], each at its own row and slot.

This is the write of a serving step’s mixed call (dew.nn.inputs.Admitted). A token whose row is past the cache’s rows, or whose slot is -1, is dropped. The cache must be full precision and unrotated, and a paged pool must have one group, as the mixed call requires.

def read_rows(rows: jax.Array) -> tuple[jax.Array, jax.Array]

Return what read returns for only rows, [len(rows), capacity, kv_heads, head_dim].

It does not dequantize, so it matches read only on a full-precision cache, which the mixed call that uses it requires.

def attends_rows(query: jax.Array, softcap: float | None) -> bool

Whether attend_rows reads this cache for query: its rows’ keys of the query’s 16-bit dtype, unquantized (the template’s decode reads no codes yet, ledger 3624), a paged pool’s pages each whole tiles or tiles of whole pages (TEMPLATE_PAGED_SUPPORTS), on a GPU its Mosaic mainloop is qualified on.

def attend_rows(
query: jax.Array,
spans,
sources: jax.Array,
softcap: float | None,
) -> jax.Array

query [B, T, H, D] through the attention template’s decode over this cache’s rows (attends_rows): batch row b over row sources[b]’s keys under spans (attention.KeySpans), read where they lie: a dense cache’s rows by sources, a paged pool’s by those rows’ tables. One call over a serving step’s work list (ledger 3623).

def case(
query: jax.Array,
*,
value_dim: int,
softcap: float | None,
arithmetic: bool,
named: str,
windowed: bool = False,
sinks: bool = False,
) -> PagedCase

The PagedCase of a decode step’s query [rows, heads, head_dim] over this pool, its values value_dim wide, under the layer’s softcap, its reference-only arithmetic (dew.nn.attention.reference_only) and its AttentionImpl, named; a windowed step reads each row from its window’s first key; a layer with sinks has attention sinks.

def decode(
query: jax.Array,
lengths: jax.Array,
softcap: float | None,
starts: jax.Array | None = None,
sinks: jax.Array | None = None,
) -> jax.Array

Attend one query per row [rows, heads, head_dim] to the first lengths slots of that row, from starts [rows] where given (a window’s first slot, which the GPU’s kernel reads), with sinks [heads] where the layer has them (the GPU’s kernel applies them).

Decode runs the paged kernel the selector takes (PAGED: the device’s, or the attention template’s decode where the GPU’s refuses the pool), which raises KernelRefused where every paged kernel’s requirements turn the step down. The TPU kernel does not scale the logits, so the query is multiplied by 1/sqrt(head_dim) first, the same scale every other attention path applies. A Mosaic kernel cannot be partitioned automatically, and neither can the GPU’s Pallas kernel, so on a mesh each runs per shard (manual_map): each holds its rows’ queries, lengths and page tables, and the key heads of the pool its query heads read, with every page, as the tables index the whole pool.