Skip to content

dew.batch

The batch contract every layer shares, the one module below them all that names it: a batch (Batch) and a variables tree (Variables), the columns a batch’s rows carry beside their fields (VALID_ROWS, EXAMPLE_ROWS) and the collection frozen weights stay in (FROZEN), how a batch is filled out with repeats of its own rows (padded_rows), how a batch’s token fields say which slots are real and where each token sits (Coordinates, real_slots), the token ids a field holds (token_ids) and which targets a language-model loss scores (scored_targets). Data builds batches by it, models read them by it, and objectives and the trainer score and place them by it, so none of them imports another for these.

NameSummary
BLOCKS_FIELDThe token field that numbers the tokens’ blocks, which the attention layers that read blocks see whole (AttentionMetadata.block_groups, prefix_blocks).
COORDINATES_FIELDThe token field that holds the tokens’ Coordinates, which a decoder reads in place of the validity field, segment_ids and positions.
EXAMPLE_ROWSA training batch’s column of each row’s place among its step’s records, int32: the order the records’ stream holds them in, whatever process read a row (dew.training.distributed.marked).
FROZENThe name of the collection where a partially trained run keeps its frozen weights.
PARAMETER_ROOTSThe collections of a variables tree that hold a model’s parameters, which a run’s loaders read and a storage dtype casts: the trained ones and the frozen.
SCAN_BLOCKSHow many blocks a running sum or maximum along a token axis cuts the axis into (prefix_sum): its scans run inside each block, so a mesh that splits the axis into up to this many shards keeps every scan on one shard, and the blocks’ carries are one reduction over the blocks before each.
TEXT_KEYBatch key the token pipeline packs [B, seq_len + 1] int32 ids under.
VALIDITY_FIELDThe token field that marks real slots.
VALID_ROWSThe field every evaluation batch holds: one bool per row, True for a real record and False for a repeat.
BatchOne batch of training examples, as the trainer and every objective read it.
CoordinatesWhere each token of a batch sits, built once (coordinates) and read by every token mixer.
Path
PathFilterA function that selects leaves of a variables tree by the tuple of dict keys above them.
VariablesA Flax variables dict: the params collection plus any other collection the modules keep (moe, batch_stats, an objective’s frozen encoders).
coordinatesThe Coordinates of [batch, length] tokens from what the batch carries: packed segments (one document a row where None), a valid mask (every token real where None), explicit positions (with segments, each document’s real tokens counted from 0 where None; without, None, each token’s place its row index or cache slot) and the packer’s max_segments (the row length where None).
dictstree (a variables tree, or a state holding several) with every container mapping a dict: a tree that holds a mapping of the kinds Flax’s unfreeze makes dicts of (a FrozenDict, a dict of another type, an OrderedDict) anywhere, under a mapping, a sequence or any other node, rebuilt over the same leaves with each one a dict, any other the caller’s own tree.
document_ranksEach slot’s count of the real tokens before it in its document [B, S], starts marking each document’s first real token but a row’s first, a padding slot taking the count before it (0 before any).
document_starts[B, S]: each real token whose segment differs from the previous real token’s in its row, a row’s first real token continuing what precedes the row and a padding slot starting nothing.
everything
field_coordinatesA row’s token fields’ layout as one Coordinates, the one conversion of a field dict’s flat layout (token_coordinates): their Coordinates as they are, else built from their flat segment_ids, attention_mask and positions, None where they carry neither; both refused.
flat_fieldThe token field name, an array, or None where the fields hold none; a Coordinates under a flat field’s name is refused.
freezeMove the params leaves that trainable rejects into the FROZEN collection.
laid_coordinatesThe Coordinates a row’s token fields carry, or None where they carry flat layout fields or none; fields carrying both are refused, as a call giving both is (token_coordinates), since one would silently win over the other, and so is anything but Coordinates under their key (ledger 1122).
leaves_by_pathEvery leaf of a variables subtree, by its dict-key path: a node that is no mapping is one leaf, whole (a quantized kernel with its scales, Qwix’s struct dataclasses a served model’s params hold).
mergeReturn a copy of tree with every leaf in overlay written over it.
owned_dictstree as a state holds its trees: every mapping a dict (dicts), the containers the state’s own, so nothing the state does to them reaches the caller’s tree.
padded_rowsThe rows count places hold of rows real ones, cycling through them, and which places hold a repeat: the rule a batch filled out with its own rows (dew.data.dataset.FilledBatch) and a request’s rows padded to its devices (dew.nn.inputs.RowPlan) both follow.
partCut the name subtree out of every collection that holds one: one module’s own tree out of an objective’s that nests several under each collection, params/policy and frozen/policy, so an adapted module’s base comes with its factors.
prefix_maxx’s running maximum along axis, cummax’s, as a program a mesh that splits the axis keeps parallel (prefix_sum).
prefix_minx’s running minimum along axis, from its end where reverse, as a program a mesh that splits the axis keeps parallel (prefix_sum).
prefix_sumx’s running sum along axis, cumsum’s, in dtype (as cumsum takes it where None: a bool’s or a narrow integer’s in the default integer), as a program a mesh that splits the axis keeps parallel: XLA’s SPMD partitioner turns one scan along a split axis into a scan with a live carry, which the GPU’s associative-scan rewrite refuses, and runs it as a loop of one iteration a token (a 2xH100 training step ran ten times slower, ledger 1905).
real_placesEach slot’s rank among its row’s real tokens under valid [B, S]: a real token’s place, a padding slot’s the rank before it, 0 before any (Coordinates.places where no positions are given).
real_slotsWhich of a request’s shape [B, S] slots hold a real token: those its validity marks, every one where it has none, less the padding a packed row numbers 0 in its documents (token_layout).
scored_targetsWhether each target the loss scores at prediction depth counts, as [B, S - 1 - depth] booleans, in xp (numpy for the host’s FLOP geometry and the data’s target counts, jax.numpy for the loss): the state at p scores the token depth + 1 later, which counts when it is not padding, when the attention mask keeps every position from the state to it, in a packed batch when the state and it share a document (segment 0 is padding), and with a role (loss_role) when roles, one per token, give the target that role.
selectReturn the subtree of tree whose leaves keep accepts, nested the same way.
shifted_spanWhich [B, S - shift] source positions can reach the token shift places later: every intervening token is valid, including both endpoints, and their document IDs match.
sown_nodesFind every node of a sown collection that holds all of markers, with its path: the module path that sowed it.
thawUndo freeze, merging the frozen leaves back into one params collection.
token_idvalue read as one token id: a Python or NumPy integer, not a bool, from 0 through int32’s largest.
token_idsvalues read as a nonempty row of token ids by token_id’s law: a list, a tuple or a one-dimensional NumPy array, as a tokenizer, a dataset’s column or a step of Dew’s own (dew.data.text.Tokenize) holds them.
token_layoutThe validity [B, S] and documents [B, S] of a row’s token fields as the decoder reads them: their Coordinates’ where the fields carry them (laid_coordinates), else the validity field and segment_ids.
token_positionsThe logical positions [B, S] of a row’s token fields as the decoder reads them: their Coordinates’ where the fields carry them (laid_coordinates), else the positions field; None where neither gives any.
token_validityThe validity [B, S] a row’s token fields hold, as given: their Coordinates’ where the fields carry them (laid_coordinates), else the validity field; None where neither holds one.
underReturn a filter that accepts the leaves below prefix.
valid_orderEach row’s real slots first, in order, then its padding slots: the physical slot behind each compact column [rows, slots], and how many slots of each row are real [rows].
with_validityfields holding valid as their validity where token_validity reads it, None removing it, their layout one Coordinates built again from it (coordinates, its documents’ starts and firsts following it), whether the fields carried coordinates or flat layout fields (ledger 1122); none where nothing is left to lay out (one document a row, every token real, at its slot), as fields that carry no layout.
withouttree without the subtree at each of paths, a path the tree does not hold left as it is.

attribute source

BLOCKS_FIELD = 'block_groups'

The token field that numbers the tokens’ blocks, which the attention layers that read blocks see whole (AttentionMetadata.block_groups, prefix_blocks).

attribute source

COORDINATES_FIELD = 'coordinates'

The token field that holds the tokens’ Coordinates, which a decoder reads in place of the validity field, segment_ids and positions.

attribute source

EXAMPLE_ROWS = 'example_rows'

A training batch’s column of each row’s place among its step’s records, int32: the order the records’ stream holds them in, whatever process read a row (dew.training.distributed.marked). The trainer’s transaction reads it for the examples’ draws (Step.example_index) and takes it out of the batch the objective sees.

attribute source

FROZEN = 'frozen'

The name of the collection where a partially trained run keeps its frozen weights.

The optimizer updates the params collection and nothing else, so the leaves that freeze keeps in params are the ones that train. The frozen leaves stay in this collection as part of the state, and thaw merges the two before the model reads them.

attribute source

PARAMETER_ROOTS = (('params',), (FROZEN,))

The collections of a variables tree that hold a model’s parameters, which a run’s loaders read and a storage dtype casts: the trained ones and the frozen.

attribute source

SCAN_BLOCKS = 64

How many blocks a running sum or maximum along a token axis cuts the axis into (prefix_sum): its scans run inside each block, so a mesh that splits the axis into up to this many shards keeps every scan on one shard, and the blocks’ carries are one reduction over the blocks before each.

attribute source

TEXT_KEY = 'text'

Batch key the token pipeline packs [B, seq_len + 1] int32 ids under.

attribute source

VALIDITY_FIELD = 'attention_mask'

The token field that marks real slots. Absent means every slot is real.

attribute source

VALID_ROWS = 'valid_rows'

The field every evaluation batch holds: one bool per row, True for a real record and False for a repeat.

A split’s last batch is padded to full size with repeats of its own rows, and a process whose share runs out before the others’ scores a copy of its last batch in which every row is a repeat. That way a pass is whole batches on every process and counts each record once. A loss weights its rows by this field, and its batch-wide terms are taken over the real rows only. Training batches never have it.

attribute source

One batch of training examples, as the trainer and every objective read it.

It is declared here and imported by dew.data.dataset, so a field added on one side has the same type on the other. The values are not all arrays. Besides the arrays a step consumes, a batch can hold the prepared text inputs of a language run, the file paths a video corpus loads lazily, the integer row counts a packer keeps and the record lists a mixture reads. A narrower value type would be wrong about what dew.data already builds.

dataclass source

class Coordinates(
positions: jax.Array | None = None,
segments: jax.Array | None = None,
valid: jax.Array | None = None,
starts: jax.Array | None = None,
firsts: jax.Array | None = None,
max_segments: int = struct.field(pytree_node=False, default=1),
right_padded: bool = struct.field(pytree_node=False, default=False),
TOKEN_ALIGNED: tuple[str, ...] = ('positions', 'segments', 'valid', 'starts'),
)

Where each token of a batch sits, built once (coordinates) and read by every token mixer.

positions [B, S] is each real token’s logical place in its document, counted over real tokens: a packed batch’s or the caller’s own, None where none is given, each token’s place then its rank among its row’s real tokens (places) or, in a cached call, the slot its cache writes it at. Rotation and NoPE’s temperature read the logical places; windows, chunks and ALiBi count each document’s real tokens from its first place (counted_places), the logical places wherever those count real tokens; a cache addresses keys by slot. segments numbers each row’s packed documents from 1, 0 on the padding slots after them, which run as a document of their own, so no layer carries a real document’s state or history into them or back; valid is the caller’s mask of real tokens; starts marks each document’s first real token but a row’s first, which continues whatever state the row holds. Each is None where it holds nothing (one document a row, every token real, no start), which a layer’s fast path reads as a static fact. firsts [B, D] are each row’s documents’ first slots in order, the row’s first slot among them, S past its last (document_firsts): the packed documents’ coordinates every varlen kernel reads (dew.nn.inputs.cu_seqlens), built with the coordinates, never by a layer: by one builder wherever coordinates are made, a host’s batch and a traced step’s alike, or their slots sliced or rotated (along_tokens; row 1923); each layer’s own scan over the batch’s flattened tokens, which a mesh splits, ran one token a loop iteration on a 2xH100 run (row 1897). max_segments bounds the documents a row holds (the packer’s static plan, else the row length) and right_padded says every row’s real tokens come first.

TOKEN_ALIGNED: tuple[str, ...]

The fields one entry a token, [B, S]; firsts is one a document, [B, D], its entries slots.

def tokens() -> dict[str, jax.Array]

The token-aligned fields it holds (TOKEN_ALIGNED), by name: what a check of a batch’s token layout reads.

def along_tokens(transform: Callable[[jax.Array], jax.Array]) -> Coordinates

transform applied along the tokens to each token-aligned field (TOKEN_ALIGNED): a slice or a rotation of the slots. The documents’ firsts, which index the slots, are built again from the moved starts (document_firsts), as every Coordinates builds them.

def places() -> jax.Array | None

Each token’s logical place in an uncached call: the given positions, else its rank among its row’s real tokens [B, S] (a padding slot’s the rank before it, 0 before any); None where every token is real and none is given, each token’s place its index. A cached call’s places are the slots its cache writes them at where none are given.

attribute source

attribute source

A function that selects leaves of a variables tree by the tuple of dict keys above them.

One filter type is used for the EMA selection, for optax.multi_transform labels and for frozen subtrees.

attribute source

A Flax variables dict: the params collection plus any other collection the modules keep (moe, batch_stats, an objective’s frozen encoders).

function source

def coordinates(
shape: tuple[int, ...] | None = None,
*,
segments=None,
valid=None,
positions=None,
max_segments: int | None = None,
right_padded: bool = False,
xp=jnp,
) -> Coordinates

The Coordinates of [batch, length] tokens from what the batch carries: packed segments (one document a row where None), a valid mask (every token real where None), explicit positions (with segments, each document’s real tokens counted from 0 where None; without, None, each token’s place its row index or cache slot) and the packer’s max_segments (the row length where None). shape is the first given field’s where None. xp is the array module the fields are built with: jax.numpy in a traced call, numpy where a data reader builds a batch’s on the host.

function source

def dicts(tree: Tree) -> Tree

tree (a variables tree, or a state holding several) with every container mapping a dict: a tree that holds a mapping of the kinds Flax’s unfreeze makes dicts of (a FrozenDict, a dict of another type, an OrderedDict) anywhere, under a mapping, a sequence or any other node, rebuilt over the same leaves with each one a dict, any other the caller’s own tree. unfreeze itself stops at a list. A record that is a mapping of its own kind (an optimizer’s nnx.State) stays as it is, its contents read the same way. The trees an objective holds, the state’s and a step’s written-back collections are dicts, so that a placement writing each placed leaf into the dict that held it (dew.distributed.host.stream) writes into the objective’s tree, and every tree of a step has the state’s structure.

function source

def document_ranks(real, starts, *, xp=jnp)

Each slot’s count of the real tokens before it in its document [B, S], starts marking each document’s first real token but a row’s first, a padding slot taking the count before it (0 before any).

function source

def document_starts(segments, valid, *, xp=jnp)

[B, S]: each real token whose segment differs from the previous real token’s in its row, a row’s first real token continuing what precedes the row and a padding slot starting nothing.

function source

def everything(path: Path) -> bool

function source

def field_coordinates(
fields: Mapping[str, jax.Array | np.ndarray | Coordinates],
shape: tuple[int, ...],
) -> Coordinates | None

A row’s token fields’ layout as one Coordinates, the one conversion of a field dict’s flat layout (token_coordinates): their Coordinates as they are, else built from their flat segment_ids, attention_mask and positions, None where they carry neither; both refused.

function source

def flat_field(
fields: Mapping[str, jax.Array | np.ndarray | Coordinates],
name: str,
) -> jax.Array | np.ndarray | None

The token field name, an array, or None where the fields hold none; a Coordinates under a flat field’s name is refused.

function source

def freeze(variables: Variables, trainable: PathFilter) -> Variables

Move the params leaves that trainable rejects into the FROZEN collection.

trainable sees full leaf paths, ("params", ...). A tree already split keeps training the leaves its params hold, beside those trainable takes from either collection, and keeps the rest frozen: an adapter attached over another keeps both training. Raises ValueError when the split keeps every leaf or none, since then there is nothing to split.

function source

def laid_coordinates(
fields: Mapping[str, jax.Array | np.ndarray | Coordinates],
) -> Coordinates | None

The Coordinates a row’s token fields carry, or None where they carry flat layout fields or none; fields carrying both are refused, as a call giving both is (token_coordinates), since one would silently win over the other, and so is anything but Coordinates under their key (ledger 1122).

function source

def leaves_by_path(tree: Variables) -> dict[tuple[str, ...], jax.Array]

Every leaf of a variables subtree, by its dict-key path: a node that is no mapping is one leaf, whole (a quantized kernel with its scales, Qwix’s struct dataclasses a served model’s params hold).

function source

def merge(tree: Variables, overlay: Variables) -> Variables

Return a copy of tree with every leaf in overlay written over it.

function source

def owned_dicts(tree: Tree) -> Tree

tree as a state holds its trees: every mapping a dict (dicts), the containers the state’s own, so nothing the state does to them reaches the caller’s tree.

function source

def padded_rows(rows: int, count: int) -> tuple[np.ndarray, np.ndarray]

The rows count places hold of rows real ones, cycling through them, and which places hold a repeat: the rule a batch filled out with its own rows (dew.data.dataset.FilledBatch) and a request’s rows padded to its devices (dew.nn.inputs.RowPlan) both follow. Places to fill from no rows are refused: there is nothing to repeat.

function source

def part(variables: Variables, name: str) -> Variables

Cut the name subtree out of every collection that holds one: one module’s own tree out of an objective’s that nests several under each collection, params/policy and frozen/policy, so an adapted module’s base comes with its factors. A step’s cotangent probes, laid out as the objective nests what its model calls sow, give name’s where it nests them under name, else pass whole (PROBES).

function source

def prefix_max(x, axis: int = 1, *, xp=jnp)

x’s running maximum along axis, cummax’s, as a program a mesh that splits the axis keeps parallel (prefix_sum).

function source

def prefix_min(x, axis: int = 1, *, reverse: bool = False, xp=jnp)

x’s running minimum along axis, from its end where reverse, as a program a mesh that splits the axis keeps parallel (prefix_sum).

function source

def prefix_sum(x, axis: int = 1, *, dtype=None, xp=jnp)

x’s running sum along axis, cumsum’s, in dtype (as cumsum takes it where None: a bool’s or a narrow integer’s in the default integer), as a program a mesh that splits the axis keeps parallel: XLA’s SPMD partitioner turns one scan along a split axis into a scan with a live carry, which the GPU’s associative-scan rewrite refuses, and runs it as a loop of one iteration a token (a 2xH100 training step ran ten times slower, ledger 1905). Here each of SCAN_BLOCKS blocks scans its own slots along an axis no mesh splits, and each block adds the sum of the blocks before it, in the result’s dtype throughout, so an integer sum wraps as cumsum’s does. Exact for integers; a float’s sum is reassociated by blocks. xp numpy computes it on the host.

function source

def real_places(valid: jax.Array) -> jax.Array

Each slot’s rank among its row’s real tokens under valid [B, S]: a real token’s place, a padding slot’s the rank before it, 0 before any (Coordinates.places where no positions are given).

function source

def real_slots(
fields: Mapping[str, jax.Array | np.ndarray | Coordinates],
shape: tuple[int, ...],
*,
xp=jnp,
)

Which of a request’s shape [B, S] slots hold a real token: those its validity marks, every one where it has none, less the padding a packed row numbers 0 in its documents (token_layout). Samplers select, place and continue a row by these, and losses score and count them, so a row padded either way reads as the row alone; a caller’s mask is narrowed, never replaced. xp is the array module.

function source

def scored_targets(
xp,
tokens,
pad_id: int | None,
attention_mask,
segment_ids,
depth: int = 0,
roles=None,
role: int | None = None,
blocks=None,
)

Whether each target the loss scores at prediction depth counts, as [B, S - 1 - depth] booleans, in xp (numpy for the host’s FLOP geometry and the data’s target counts, jax.numpy for the loss): the state at p scores the token depth + 1 later, which counts when it is not padding, when the attention mask keeps every position from the state to it, in a packed batch when the state and it share a document (segment 0 is padding), and with a role (loss_role) when roles, one per token, give the target that role. With blocks, each token’s block (-1 none), a target in its state’s block does not count. Blocks are contiguous runs (AttentionMetadata.block_groups), so this is exact for the trunk’s targets (depth 0): a prediction depth reads blocks beyond its own source, so token_scores refuses depths with blocks.

function source

def select(tree: Variables, keep: PathFilter) -> Variables

Return the subtree of tree whose leaves keep accepts, nested the same way.

A branch with no accepted leaf is dropped, so the result is what the EMA stores and what merge puts back. Raises ValueError when keep accepts no leaf at all.

function source

def shifted_span(
shape: tuple[int, ...],
shift: int,
*,
valid=None,
segments=None,
xp=jnp,
)

Which [B, S - shift] source positions can reach the token shift places later: every intervening token is valid, including both endpoints, and their document IDs match. Padding IDs and target roles belong to the caller; segments describes identity, not validity. xp is NumPy for host geometry and JAX NumPy for model/loss execution.

function source

def sown_nodes(tree, *markers: str = ()) -> list[tuple[tuple[str, ...], Mapping]]

Find every node of a sown collection that holds all of markers, with its path: the module path that sowed it.

A sown collection nests by module path, so what a layer sowed is the branch that names the keys it sowed under. The search stops at such a branch rather than descending into the arrays below it, and walks the children it does descend into in sorted key order.

function source

def thaw(variables: Variables) -> Variables

Undo freeze, merging the frozen leaves back into one params collection.

The flags of a packed group’s frozen members (PACKED_FROZEN) go with the split they describe. Variables without either collection are returned unchanged, and a part that trains nothing (part of a frozen module) holds its frozen leaves alone.

function source

def token_id(value: object, *, where: str) -> int

value read as one token id: a Python or NumPy integer, not a bool, from 0 through int32’s largest. Every parser of ids, a dataset’s or a request’s, reads them by this law; where names what held the value.

function source

def token_ids(values: object, *, where: str) -> list[int]

values read as a nonempty row of token ids by token_id’s law: a list, a tuple or a one-dimensional NumPy array, as a tokenizer, a dataset’s column or a step of Dew’s own (dew.data.text.Tokenize) holds them.

function source

def token_layout(fields: Mapping[str, jax.Array | np.ndarray | Coordinates], *, xp=jnp)

The validity [B, S] and documents [B, S] of a row’s token fields as the decoder reads them: their Coordinates’ where the fields carry them (laid_coordinates), else the validity field and segment_ids. Each is None where nothing gives it (every slot real, one document a row). xp is the array module.

function source

def token_positions(
fields: Mapping[str, jax.Array | np.ndarray | Coordinates],
*,
xp=jnp,
)

The logical positions [B, S] of a row’s token fields as the decoder reads them: their Coordinates’ where the fields carry them (laid_coordinates), else the positions field; None where neither gives any. xp is the array module.

function source

def token_validity(fields: Mapping[str, jax.Array | np.ndarray | Coordinates])

The validity [B, S] a row’s token fields hold, as given: their Coordinates’ where the fields carry them (laid_coordinates), else the validity field; None where neither holds one.

function source

def under(*prefix: str = ()) -> PathFilter

Return a filter that accepts the leaves below prefix.

For example, under("params", "context_encoder").

function source

def valid_order(valid: jax.Array, *, xp=jnp) -> tuple[jax.Array, jax.Array]

Each row’s real slots first, in order, then its padding slots: the physical slot behind each compact column [rows, slots], and how many slots of each row are real [rows]. A stable sort of the validity, so a row’s tokens keep their order; a caller says what its columns past the count hold (decoding’s -1 history tokens, a convolution’s zero column). xp is the array module.

function source

def with_validity(
fields: Mapping[str, jax.Array | np.ndarray | Coordinates],
valid: jax.Array | np.ndarray | None,
) -> dict

fields holding valid as their validity where token_validity reads it, None removing it, their layout one Coordinates built again from it (coordinates, its documents’ starts and firsts following it), whether the fields carried coordinates or flat layout fields (ledger 1122); none where nothing is left to lay out (one document a row, every token real, at its slot), as fields that carry no layout. A field dict’s layout so takes one shape for one set of given fields, which a pool’s processes agree on (dew.nn.inputs.agreed_validity). Arrays stay on the host where valid and the fields are numpy.

function source

def without(tree: Variables, paths: Sequence[Sequence[str]]) -> Variables

tree without the subtree at each of paths, a path the tree does not hold left as it is.