Skip to content

dew.inference.banks

Build and place parameter banks for generation, one bank at a time.

A scanned run of identical layers reads each parameter as one array with the layer axis in front; that stacked array is a bank, and one layer’s slice of it is a row. (dew.nn.backbones.decoder_stack.StackView adapts banks to the layers_N subtrees a checkpoint stores.) This module builds the banked store directly. It reads, stacks and places each bank separately and releases that bank’s rows before reading the next, so a stack that does not fit in device memory is never held as separate layers and as banks at the same time.

Layout.host_parameters puts banks in pinned host memory, and the stack fetches them one layer at a time (run_stack). Everything else is placed the way the trainer places a parameter. A LayerBanks source reads from a run directory (CheckpointBanks), a tree in memory (HeldBanks) or local HF safetensors (SafetensorsBanks). A source’s place puts whole banks on the mesh. SafetensorsBanks.stream leaves the decoder weights on disk instead, and the prefetch loop reads them during execution.

NameSummary
CheckpointBanksServes banks by restoring a run’s published weights one bank at a time.
DiskBankStatsCumulative host-read statistics of a SafetensorsBanks source, as its stats() returns them.
HeldBanksServes banks from a variables tree already held in memory.
LayerBanksServes a banked store’s values, read one bank at a time.
SafetensorsBanksReads a local HF decoder checkpoint without materializing its layer stack.
StreamedBankFetches one run’s layers from a SafetensorsBanks source during execution.

dataclass source

class CheckpointBanks(directory: str, step: int | None = None, ema: bool = False)

Serves banks by restoring a run’s published weights one bank at a time.

With ema, the averaged copy is merged over the live weights, as Checkpoints.variables does, so a bank holds what the run publishes. step selects a checkpoint. When it is None, the source resolves it to the latest checkpoint once, when the source is built, so every bank of one load comes from the same checkpoint. A directory without a checkpoint raises FileNotFoundError.

Each row is restored to device memory and written into its bank, placed where the store places it, before the next is read. So a load holds a bank and one row beside it, never the whole model twice: a stack of one run, a bank of every layer, would otherwise hold its rows beside it.

stored: Variables

The shape and dtype trees the checkpoint at step holds, read once when the source is built.

def shapes() -> Variables
def entry(placement: Placement) -> Variables
def bank(
layers: Sequence[int],
placement: Placement,
*,
namespace: tuple[str, ...] = (),
) -> Variables
def place(
model: BankedStacks,
*,
mesh: MeshSpec | Mesh | None = None,
layout: Layout | None = None,
) -> Variables

LayerBanks.place, every entry and bank of the load read through one reader of the run, so the checkpoint’s metadata is read once a load, not once a layer.

dataclass source

class DiskBankStats(
cache_bytes: int,
hits: int,
misses: int,
bytes_read: int,
read_seconds: float,
wait_seconds: float,
)

Cumulative host-read statistics of a SafetensorsBanks source, as its stats() returns them.

cache_bytes is the size of the layers cached now. hits and misses count layer reads served from the cache and from disk, and bytes_read is the bytes read on misses. read_seconds is the time spent reading, which includes layout conversion and can overlap compute. wait_seconds is the time callers of read waited for a row.

dataclass source

class HeldBanks(variables: Variables)

Serves banks from a variables tree already held in memory.

variables is the tree that model.init or a loader produced, with one subtree per layer. The source borrows its arrays and never donates or deletes them, so the caller’s tree stays usable. A load therefore holds the source and the growing banked store at once, which costs two copies of the model. Use CheckpointBanks for weights that do not fit twice.

def shapes() -> Variables
def entry(placement: Placement) -> Variables
def bank(
layers: Sequence[int],
placement: Placement,
*,
namespace: tuple[str, ...] = (),
) -> Variables

class source

class LayerBanks(Protocol)

Serves a banked store’s values, read one bank at a time.

Paths are canonical below each collection, and a namespace selects a decoder inside a wrapper model. The memory a source needs to hold one bank depends on the source, and it bounds what a load costs. A source subclasses this class to inherit place, which builds a model’s banked store from the source.

def shapes() -> Variables

Return the whole store as jax.ShapeDtypeStruct leaves, one per layer.

def entry(placement: Placement) -> Variables

Read the canonical leaves that placement selects, and only those.

def bank(
layers: Sequence[int],
placement: Placement,
*,
namespace: tuple[str, ...] = (),
) -> Variables

Return the bank of one run of layers, on the shardings or Formats in placement.

The layers are stacked in order on a new leading axis; a run of one layer returns that layer’s own subtree. A Format also sets a physical layout, which the source must preserve. The returned collections are local to namespace.

def place(
model: BankedStacks,
*,
mesh: MeshSpec | Mesh | None = None,
layout: Layout | None = None,
) -> Variables

Build, from these weights, the banked store that model’s runs read.

It waits for each bank’s copies to finish before reading the next bank, so a load has one bank’s transfers in flight and costs the store plus what this source holds. It does not donate or delete any of the source’s arrays.

layout.host_parameters may name only layers of the stack, because the stack fetches each layer when it reaches it. An embedding or head in host memory would be copied over whole and cost the device memory it was meant to save. A run whose layers the layout’s patterns place differently is refused as well, because a bank is one array with one sharding. Both checks raise ValueError before anything is read.

class source

class SafetensorsBanks(
directory: str | Path,
*,
cache_bytes: int = 0,
param_dtype: DTypeLike | Literal['auto'] = 'auto',
read_ahead: bool = True,
)

Reads a local HF decoder checkpoint without materializing its layer stack.

The checkpoint is a decoder, a media wrapper around one, or the native backbone a release with a separately stored readout describes (a slot decision release). It translates the weights as Pretrained.load does (dew.interop.pretrained.native_backbone), with its SourceLeaf recipes over read-only memory maps, including expert stacking, transposition and the cast to param_dtype (by default the checkpoint’s dtype). Quantized checkpoints are refused, because their codec may dequantize the entire checkpoint before translation, which would break the memory bound below.

The cache keeps complete layers in read order until cache_bytes is full, and holds them until close. A sequential decoder visits every layer for each token, so an LRU cache would evict every layer once the model is larger than the cache; keeping a fixed prefix of layers makes a partial cache useful. One read-ahead slot holds the next layer, even with a zero cache, unless read_ahead is False. Host memory for weights is bounded by cache_bytes, two rows and one leaf’s conversion scratch. Embeddings, heads, a media wrapper’s towers and projectors (read whole, as the wrapper’s reader reads them) and the KV cache stay resident outside that bound, and the bound does not count the kernel’s page cache.

Keep the source open and its files unchanged until all executions finish, because a compiled callback cannot read a closed source.

def close() -> None
def stats() -> DiskBankStats
def shapes() -> Variables
def layer_bytes(index: int, *, namespace: tuple[str, ...] = ()) -> int
def entry(placement: Placement) -> Variables
def bank(
layers: Sequence[int],
placement: Placement,
*,
namespace: tuple[str, ...] = (),
) -> Variables
def stream(
model: BankedStacks,
*,
mesh: MeshSpec | Mesh | None = None,
layout: Layout | None = None,
) -> Variables

Return variables whose decoder layers the banked inference loop reads from disk, on one device.

Only the non-decoder leaves are loaded now. Each declared run gets a static StreamedBank handle in the runtime-only streaming collection, and the stack’s prefetch, which holds two rows at a time, calls it during execution.

This works for single-device inference only. A mesh of more than one device, host placement in layout, or a stack without scan_layers raises ValueError before any weight is loaded. The host callback needs the CPU backend beside the accelerator, so set JAX_PLATFORMS=cuda,cpu if you set platforms explicitly. Keep this source open until executions complete.

def read(
index: int,
*,
namespace: tuple[str, ...] = (),
following: int | None = None,
) -> Variables

Read the row of layer index and start reading the following layer before returning.

The read-ahead happens only when following is given and the source’s read_ahead is on. Reads from executions that share this source run one at a time. A pending read-ahead finishes before a different request is served, so concurrent calls cannot build up an unbounded queue of rows.

dataclass source

class StreamedBank(
source: SafetensorsBanks,
first: int,
count: int,
namespace: tuple[str, ...],
device: SingleDeviceSharding,
following: int | None,
)

Fetches one run’s layers from a SafetensorsBanks source during execution.

It is a runtime-only handle, with no parameter buffer and no serialized model field. The handle is a static pytree node compared by identity, so changing the source retraces and changing the tokens does not. The callback has side effects, so JAX neither folds it at trace time nor skips it when an executable is reused.

def shapes() -> Variables
def fetch(index, dependency) -> Variables