Skip to content

dew.data.mixing

Accepted-row accounting and mixtures measured in valid target tokens.

NameSummary
COUNTA CorpusCounts counter that counts (counters): a number a position keeps as it is.
KINDSA CorpusCounts counter by kind (counters): a Counter a position keeps as a record of its kinds.
CorpusCountsWhat one corpus’s read counted, under its identity (name, which plans and saved positions key it by) and the short name its metrics go by (label).
COUNTER
countersCorpusCounts’ counters, declared once with their codec (COUNT, KINDS): what a saved position keeps of a corpus’s counts (state, restore) and what two reads’ counts add (add).
CUT_COUNTSA corpus’s counts by kind but its rejections, which a position version 7 saved kept alone (dew.data.plan.earlier_position).
ChunkedA chunk expansion counted at its fan-out, with counters in the iterator position: its documents and their chunks, and no row (CorpusCounts.rows), which the counters about it count.
Carriedparent’s rows, each marker’s source examples (Rejected.consumed) carried on to the next row and the markers dropped, counting nothing: the carrier after a step whose markers no corpus’s count follows, a batched map’s group it makes no rows of (dew.data.lowering._Groups).
AcceptedGrain’s filter, with checkpointed counts.
token_weightsWindow shares for authoritative per-corpus valid-target counts.
ProportionsA weighted mix’s selection policy, Grain’s (MapDataset.mix): the weights scaled so the least is 100 and truncated to whole proportions, and position k of the mix read from the corpus whose count of records a prefix holds grows from k to k + 1.
ProportionalSelectionGrain’s weighted mix of corpora of lengths records as a selection map (MapDataset.select_from_datasets), by Proportions, its corpus choices permuted within each window by seed where one is given: as long as the mix reads no record of any corpus twice.
TokenDeficitPick the corpus furthest below its token share, with index-order ties.
KeyedMixGrain’s weighted mix of corpora, each read as it is drawn (_MixReads), and each keyed by its member’s key (keys): Grain saves a keyed mix’s positions by key and restores those it finds, so a member that joins starts at its first row and one that left is dropped while the others resume where they were.
rewoundstart, the position a repeat rewinds its pass to, with each counter’s counts as current holds them (COUNTER: its counts, run of rejections and warning): a pass read again goes on counting the corpus, where a restored position replaces its counts.
state_countsRead per-corpus counters through Grain’s nested iterator checkpoints.
data_metricsThe metrics a run logs of its corpora (CorpusCounts.metrics), each under its label; corpora that share one are numbered in the order of their identities, past every label a corpus has.

attribute source

COUNT = {'count': 'number'}

A CorpusCounts counter that counts (counters): a number a position keeps as it is.

attribute source

KINDS = {'count': 'kinds'}

A CorpusCounts counter by kind (counters): a Counter a position keeps as a record of its kinds.

dataclass source

class CorpusCounts(
name: str,
label: str,
rows: bool = True,
examples: bool = False,
attempted: int = 0,
accepted: int = 0,
truncated: int = 0,
delivered_rows: int = 0,
chunks: int = 0,
chunk_records: int = 0,
rejected: Counter[str] = Counter(),
cut_examples: Counter[str] = Counter(),
cut_rows: Counter[str] = Counter(),
cut_removed: Counter[str] = Counter(),
left_out: Counter[str] = Counter(),
rejected_cut: Counter[str] = Counter(),
left_out_cut: Counter[str] = Counter(),
declined: Counter[str] = Counter(),
tokens: int = 0,
)

What one corpus’s read counted, under its identity (name, which plans and saved positions key it by) and the short name its metrics go by (label). accepted counts the rows it delivered and truncated those of them cut short, its source examples where an objective makes several rows of each (FlatMap.examples): in a run bounded in examples, those whose last rows it delivered, as each row says (examples), an example a step refused or made no row of counted as refused; in any other, those it read before the objective’s step made rows of them; cut_examples, cut_rows and cut_removed count, by what each cut (a decision row’s instructions, options, state, or a context it did not read), the delivered rows a cut shortened, the rows it shortened of them, and the tokens it removed (characters, for a kind that names them: state_characters), as the delivered rows report them (delivered); a rejection counts what the step that drops it reads, a record or a piece or window a step made of one, a member a delivered row left out among them. declined counts, by the reason its layout refuses it, the delivered rows whose augmentation a training read declines where it draws it (a decision row’s “none of the above”, _dew_declined), each then read as it was admitted. attempted counts what its last filter read, for its own warning; it describes no one unit of the corpus, so no metric shows it. A counter past an objective’s step that makes rows of the source examples it counted before (rows false) counts the rows it delivers, cut short and cut, and what it rejects of them, but no example: a corpus none of whose counters counts its examples shows no data/rows. delivered_rows counts the rows it delivered, which the share of them cut short is of: accepted’s count where each row is one, not where a row stands for the examples it completes (examples) or for none. A mixture’s members count their records as the mix draws them; one a run bounds in examples counts each member’s where their rows are delivered, and what a step after the mix refuses of a member’s rows under that member (MEMBER, MEMBERS): a step of Grain’s own after the mix, which no row’s member goes through, refuses under the mixture’s name.

examples: bool

Whether the rows it counts delivered stand for the corpus’s source examples their last rows complete, as each says (CONSUMED, less those it carries for examples a step refused or made no row of, REFUSED): a run bounded in examples counts them so where its rows are delivered, in the reading process (dew.data.lowering._Build.records).

left_out: Counter[str]

The members delivered rows left out, by reason (_dew_rejected), counted apart from the records the corpus rejected whole, so a recount of the delivered rows (counted_anew) starts them over.

rejected_cut: Counter[str]

The tokens the cuts of the records the corpus rejected whole removed, by what each cut (Rejected.cut), counted with their rejections.

left_out_cut: Counter[str]

The tokens the cuts of the members delivered rows left out removed, by what each cut (_dew_left_out_cut), counted with them.

tokens: int

The valid target tokens of the rows its reader weighed as it accepted them, under its member where a mixture’s rows are (row 4119): what a mixture measured in tokens (mix(..., by="tokens")) drew of each member, the shares it balances, or a layout’s windows that hold a target. A step after the weighing that refuses a weighed row counts it under data/rejected, its tokens staying weighed.

refused: Counter[str]

What the corpus left out, by reason: the records it rejected whole and the members its delivered rows left out (left_out).

refused_cut: Counter[str]

The tokens the cuts of what the corpus left out removed, by what each cut: of the records it rejected whole and of the members its delivered rows left out.

def delivered(
element: Batch | bytes,
examples: int | None = None,
tokens: int | None = None,
) -> None

Count one row the corpus delivered: accepted, cut short where it says it was (_dew_truncated), each cut it reports (_dew_cut: by what it cut, the rows it shortened and the tokens it removed), the members it left out by reason (_dew_rejected) and the tokens their cuts removed (_dew_left_out_cut), and the augmentation a read declines for it, by reason (_dew_declined). A mixture’s counter counts the examples of this corpus the row completes (MEMBERS), and a counter that weighs rows its valid target tokens.

def dropped(rejected: Rejected) -> None

Count one record the corpus rejected whole, under its reason, with the tokens its cuts removed (Rejected.cut).

def counted_anew() -> None

Start over the counts of delivered rows, for a read that counts them again whole (a memory read’s rows, read once more as it holds them).

def metrics(label: str) -> dict[str, float]

The metrics a run logs of this corpus under label, each exact in one unit: the rows it delivered, its rejections and the augmentation it declined by reason as integers, the share of those rows cut short and the chunks per chunked record.

def state() -> Mapping[str, object]

Its counters (counters) as a saved position keeps them.

def restore(state: Mapping[str, object]) -> None

Its counters read back from a saved position (state), each by its codec.

def add(counts: CorpusCounts) -> None

Add counts, another iterator’s of the records it read, to its counters.

def followed(reader: CorpusCounts) -> CorpusCounts

These counts of the prefix a memory read ran, followed by reader’s of the stages it read after them: the reader delivered the rows and counted those cut short, and its examples where it counts them (rows), else the prefix did, and the two dropped and chunked apart; the tokens are the first weighing’s, whichever weighed.

def expanded(outputs: Sequence[Record]) -> None

Count one chunked document and its fan-out before later stages can reject chunks.

attribute source

COUNTER = 'dew.corpus_counts'

function source

def counters() -> tuple[dataclasses.Field, ...]

CorpusCounts’ counters, declared once with their codec (COUNT, KINDS): what a saved position keeps of a corpus’s counts (state, restore) and what two reads’ counts add (add).

attribute source

CUT_COUNTS = tuple(field.name for field in counters() if field.metadata == KINDS and field.name != 'rejected')

A corpus’s counts by kind but its rejections, which a position version 7 saved kept alone (dew.data.plan.earlier_position).

class source

class Chunked(
parent: pygrain.IterDataset[Record],
name: str,
label: str,
expand: Callable[[CorpusCounts], gx.FlatMapTransform],
)

A chunk expansion counted at its fan-out, with counters in the iterator position: its documents and their chunks, and no row (CorpusCounts.rows), which the counters about it count.

class source

class Carried(parent: pygrain.IterDataset[Batch | Rejected])

parent’s rows, each marker’s source examples (Rejected.consumed) carried on to the next row and the markers dropped, counting nothing: the carrier after a step whose markers no corpus’s count follows, a batched map’s group it makes no rows of (dew.data.lowering._Groups). The examples it owes are checkpointed with its parent’s position; those the stream ends owing are its end’s (dew.training._fit.drained).

class source

class Accepted(
parent: pygrain.IterDataset[Batch | Rejected] | pygrain.IterDataset[Batch],
name: str,
label: str,
*,
tokens: Callable[[Batch], int] | None = None,
certain_after: int | None = None,
rows: bool = True,
examples: bool = False,
members: Sequence[tuple[str, str, str]] = (),
credits: bool = False,
)

Grain’s filter, with checkpointed counts. A filter of an endless repeat whose passes it rejects alike refuses a run of certain_after rejections, two whole passes, as reading nothing forever; any other run of rejections it logs and reads on, a fetch that failed for now say. One past an objective’s step that makes rows of source examples counts none of the rows it keeps (rows false, CorpusCounts.rows). A mixture’s counter past the mix counts what it rejects of each member’s rows under that member (members, each’s name and label), and its last counter, which credits them, the examples each member’s rows complete as it delivers them (MEMBERS); each member is its key, name and label.

function source

def token_weights(
names: Sequence[str],
weights: Sequence[float],
windows: Sequence[int],
targets: Sequence[int],
) -> tuple[float, ...]

Window shares for authoritative per-corpus valid-target counts.

dataclass source

class Proportions(proportions: tuple[int, ...], seed: int | None = None)

A weighted mix’s selection policy, Grain’s (MapDataset.mix): the weights scaled so the least is 100 and truncated to whole proportions, and position k of the mix read from the corpus whose count of records a prefix holds grows from k to k + 1. That schedule repeats every window of the mix, its proportions’ sum of positions, in which each corpus reads its proportion; a seed permutes the corpus choices within each window by a permutation of its own (window), so the mix reads its corpora in no fixed order and every window still holds each at its share. A phase counts its prefix by it (counts) as the mix reads by it (ProportionalSelection), so the two never disagree.

def of(weights: Sequence[float], seed: int | None = None) -> Proportions
def window(index: int) -> tuple[np.ndarray, np.ndarray]

The indexth window of the mix: the corpus each of its positions reads, and how many of that corpus’s records the window read before it. Unseeded, every window is Grain’s schedule; seeded, window index is it permuted by np.random.default_rng([seed, index]), as a shard order’s epochs are (dew.data.ordering).

def counts(records: int) -> tuple[int, ...]

How many records of each corpus the first records positions of the mix read.

def covering(sizes: Sequence[int]) -> int

The fewest positions of the mix in which every corpus reads sizes of its records, its size: the pass in which each repeated corpus has come round once.

def at(position: int) -> tuple[int, int]

The corpus position position of the mix reads, the one whose count grows with it, and which of that corpus’s records: the next it hasn’t read.

def run(start: int, positions: int) -> tuple[np.ndarray, np.ndarray] | None

at of positions positions in a row from start, as two arrays, corpus and record: counts’ split taken for every position at once, a corpus at a time, so a run costs what one position does in Python and a corpus’s arithmetic in its vector. None where a position’s products would leave 64-bit integers, which at reads exactly.

class source

class ProportionalSelection(
lengths: Sequence[int],
weights: Sequence[float],
seed: int | None = None,
)

Grain’s weighted mix of corpora of lengths records as a selection map (MapDataset.select_from_datasets), by Proportions, its corpus choices permuted within each window by seed where one is given: as long as the mix reads no record of any corpus twice.

RUN

The positions read ahead together once a reader reads one after another (Proportions.run).

class source

class TokenDeficit(
parents: Sequence[pygrain.IterDataset[Batch]],
weights: Sequence[float],
keys: Sequence[str],
*,
tokens: Callable[[Batch], int],
names: Sequence[str],
labels: Sequence[str],
)

Pick the corpus furthest below its token share, with index-order ties.

class source

class KeyedMix(
parents: Sequence[pygrain.IterDataset[Batch]],
weights: Sequence[float],
keys: Sequence[str],
seed: int | None = None,
)

Grain’s weighted mix of corpora, each read as it is drawn (_MixReads), and each keyed by its member’s key (keys): Grain saves a keyed mix’s positions by key and restores those it finds, so a member that joins starts at its first row and one that left is dropped while the others resume where they were. With a seed, the corpus each position reads is Proportions(seed=...)’s, its windows’ choices permuted (_SeededMix), its position saved alike.

function source

def rewound(
start: Mapping[str, object],
current: Mapping[str, object],
) -> Mapping[str, object]

start, the position a repeat rewinds its pass to, with each counter’s counts as current holds them (COUNTER: its counts, run of rejections and warning): a pass read again goes on counting the corpus, where a restored position replaces its counts. Every other part of a position is the iterator’s own and is left as start has it. A mix’s corpus the pass had not opened at its start (_MixRead’s _UNOPENED) is rewound to that, carrying the position it holds now, open or not, whose counters, every one its stages keep, it takes on as it opens again.

function source

def state_counts(state: Mapping[str, object]) -> dict[str, CorpusCounts]

Read per-corpus counters through Grain’s nested iterator checkpoints.

Within one iterator a corpus’s counters are its stages’: the outermost counts the rows it delivered and those of them cut short, and each its own rejections. The iterators Grain interleaves, its worker processes or a corpus’s shards, each read records of their own, so their counts add, a shard waiting past the cycle (a future state) included.

function source

def data_metrics(counts: Iterable[CorpusCounts]) -> dict[str, float]

The metrics a run logs of its corpora (CorpusCounts.metrics), each under its label; corpora that share one are numbered in the order of their identities, past every label a corpus has.