dew.data.mixing
Accepted-row accounting and mixtures measured in valid target tokens.
| Name | Summary |
|---|---|
COUNT | A CorpusCounts counter that counts (counters): a number a position keeps as it is. |
KINDS | A CorpusCounts counter by kind (counters): a Counter a position keeps as a record of its kinds. |
CorpusCounts | What one corpus’s read counted, under its identity (name, which plans and saved positions key it by) and the short name its metrics go by (label). |
COUNTER | |
counters | CorpusCounts’ counters, declared once with their codec (COUNT, KINDS): what a saved position keeps of a corpus’s counts (state, restore) and what two reads’ counts add (add). |
CUT_COUNTS | A corpus’s counts by kind but its rejections, which a position version 7 saved kept alone (dew.data.plan.earlier_position). |
Chunked | A chunk expansion counted at its fan-out, with counters in the iterator position: its documents and their chunks, and no row (CorpusCounts.rows), which the counters about it count. |
Carried | parent’s rows, each marker’s source examples (Rejected.consumed) carried on to the next row and the markers dropped, counting nothing: the carrier after a step whose markers no corpus’s count follows, a batched map’s group it makes no rows of (dew.data.lowering._Groups). |
Accepted | Grain’s filter, with checkpointed counts. |
token_weights | Window shares for authoritative per-corpus valid-target counts. |
Proportions | A weighted mix’s selection policy, Grain’s (MapDataset.mix): the weights scaled so the least is 100 and truncated to whole proportions, and position k of the mix read from the corpus whose count of records a prefix holds grows from k to k + 1. |
ProportionalSelection | Grain’s weighted mix of corpora of lengths records as a selection map (MapDataset.select_from_datasets), by Proportions, its corpus choices permuted within each window by seed where one is given: as long as the mix reads no record of any corpus twice. |
TokenDeficit | Pick the corpus furthest below its token share, with index-order ties. |
KeyedMix | Grain’s weighted mix of corpora, each read as it is drawn (_MixReads), and each keyed by its member’s key (keys): Grain saves a keyed mix’s positions by key and restores those it finds, so a member that joins starts at its first row and one that left is dropped while the others resume where they were. |
rewound | start, the position a repeat rewinds its pass to, with each counter’s counts as current holds them (COUNTER: its counts, run of rejections and warning): a pass read again goes on counting the corpus, where a restored position replaces its counts. |
state_counts | Read per-corpus counters through Grain’s nested iterator checkpoints. |
data_metrics | The metrics a run logs of its corpora (CorpusCounts.metrics), each under its label; corpora that share one are numbered in the order of their identities, past every label a corpus has. |
COUNT = {'count': 'number'}A CorpusCounts counter that counts (counters): a number a position keeps as it is.
KINDS = {'count': 'kinds'}A CorpusCounts counter by kind (counters): a Counter a position keeps as a record of its kinds.
CorpusCounts
Section titled “CorpusCounts”class CorpusCounts( name: str, label: str, rows: bool = True, examples: bool = False, attempted: int = 0, accepted: int = 0, truncated: int = 0, delivered_rows: int = 0, chunks: int = 0, chunk_records: int = 0, rejected: Counter[str] = Counter(), cut_examples: Counter[str] = Counter(), cut_rows: Counter[str] = Counter(), cut_removed: Counter[str] = Counter(), left_out: Counter[str] = Counter(), rejected_cut: Counter[str] = Counter(), left_out_cut: Counter[str] = Counter(), declined: Counter[str] = Counter(), tokens: int = 0,)What one corpus’s read counted, under its identity (name, which plans and saved positions
key it by) and the short name its metrics go by (label). accepted counts the rows it
delivered and truncated those of them cut short, its source examples where an objective makes
several rows of each (FlatMap.examples): in a run bounded in examples, those whose last rows it
delivered, as each row says (examples), an example a step refused or made no row of counted as
refused; in any other, those it read before the objective’s step made rows of them; cut_examples,
cut_rows and
cut_removed count, by what each cut (a decision row’s instructions, options, state, or a context it
did not read), the delivered rows a cut shortened, the rows it shortened of them, and the tokens it
removed (characters, for a kind that names them: state_characters), as the delivered rows report
them (delivered); a rejection counts what the step that
drops it reads, a record or a piece or window a step made of one, a member a delivered row left out
among them. declined counts, by the reason its layout refuses it, the delivered rows whose
augmentation a training read declines where it draws it (a decision row’s “none of the above”,
_dew_declined), each then read as it was admitted. attempted counts what its last
filter read, for its own warning; it describes no one unit of the corpus, so no metric shows it.
A counter past an objective’s step that makes rows of the source examples it counted before
(rows false) counts the rows it delivers, cut short and cut, and what it rejects of them, but no
example: a corpus none of whose counters counts its examples shows no data/rows. delivered_rows
counts the rows it delivered, which the share of them cut short is of: accepted’s count where each
row is one, not where a row stands for the examples it completes (examples) or for none. A
mixture’s members count their records as the mix draws them; one a run bounds in examples counts each
member’s where their rows are delivered, and what a step after the mix refuses of a member’s rows
under that member (MEMBER, MEMBERS): a step of Grain’s own after the mix, which no row’s member
goes through, refuses under the mixture’s name.
examples: bool-
Whether the rows it counts delivered stand for the corpus’s source examples their last rows complete, as each says (
CONSUMED, less those it carries for examples a step refused or made no row of,REFUSED): a run bounded in examples counts them so where its rows are delivered, in the reading process (dew.data.lowering._Build.records). left_out: Counter[str]-
The members delivered rows left out, by reason (
_dew_rejected), counted apart from the records the corpus rejected whole, so a recount of the delivered rows (counted_anew) starts them over. rejected_cut: Counter[str]-
The tokens the cuts of the records the corpus rejected whole removed, by what each cut (
Rejected.cut), counted with their rejections. left_out_cut: Counter[str]-
The tokens the cuts of the members delivered rows left out removed, by what each cut (
_dew_left_out_cut), counted with them. tokens: int-
The valid target tokens of the rows its reader weighed as it accepted them, under its member where a mixture’s rows are (row 4119): what a mixture measured in tokens (
mix(..., by="tokens")) drew of each member, the shares it balances, or a layout’s windows that hold a target. A step after the weighing that refuses a weighed row counts it underdata/rejected, its tokens staying weighed. refused: Counter[str]-
What the corpus left out, by reason: the records it rejected whole and the members its delivered rows left out (
left_out). refused_cut: Counter[str]-
The tokens the cuts of what the corpus left out removed, by what each cut: of the records it rejected whole and of the members its delivered rows left out.
CorpusCounts.delivered
Section titled “CorpusCounts.delivered”def delivered( element: Batch | bytes, examples: int | None = None, tokens: int | None = None,) -> NoneCount one row the corpus delivered: accepted, cut short where it says it was
(_dew_truncated), each cut it reports (_dew_cut: by what it cut, the rows it shortened and
the tokens it removed), the members it left out by reason (_dew_rejected) and the tokens their
cuts removed (_dew_left_out_cut), and the augmentation a read declines for it, by reason
(_dew_declined). A mixture’s counter counts the examples of this corpus the row completes
(MEMBERS), and a counter that weighs rows its valid target tokens.
CorpusCounts.dropped
Section titled “CorpusCounts.dropped”def dropped(rejected: Rejected) -> NoneCount one record the corpus rejected whole, under its reason, with the tokens its cuts
removed (Rejected.cut).
CorpusCounts.counted_anew
Section titled “CorpusCounts.counted_anew”def counted_anew() -> NoneStart over the counts of delivered rows, for a read that counts them again whole (a memory read’s rows, read once more as it holds them).
CorpusCounts.metrics
Section titled “CorpusCounts.metrics”def metrics(label: str) -> dict[str, float]The metrics a run logs of this corpus under label, each exact in one unit: the rows it
delivered, its rejections and the augmentation it declined by reason as integers, the share
of those rows cut short and the chunks per chunked record.
CorpusCounts.state
Section titled “CorpusCounts.state”def state() -> Mapping[str, object]Its counters (counters) as a saved position keeps them.
CorpusCounts.restore
Section titled “CorpusCounts.restore”def restore(state: Mapping[str, object]) -> NoneIts counters read back from a saved position (state), each by its codec.
CorpusCounts.add
Section titled “CorpusCounts.add”def add(counts: CorpusCounts) -> NoneAdd counts, another iterator’s of the records it read, to its counters.
CorpusCounts.followed
Section titled “CorpusCounts.followed”def followed(reader: CorpusCounts) -> CorpusCountsThese counts of the prefix a memory read ran, followed by reader’s of the stages it
read after them: the reader delivered the rows and counted those cut short, and its examples
where it counts them (rows), else the prefix did, and the two dropped and chunked apart; the
tokens are the first weighing’s, whichever weighed.
CorpusCounts.expanded
Section titled “CorpusCounts.expanded”def expanded(outputs: Sequence[Record]) -> NoneCount one chunked document and its fan-out before later stages can reject chunks.
COUNTER
Section titled “COUNTER”COUNTER = 'dew.corpus_counts'counters
Section titled “counters”def counters() -> tuple[dataclasses.Field, ...]CorpusCounts’ counters, declared once with their codec (COUNT, KINDS): what a saved position
keeps of a corpus’s counts (state, restore) and what two reads’ counts add (add).
CUT_COUNTS
Section titled “CUT_COUNTS”CUT_COUNTS = tuple(field.name for field in counters() if field.metadata == KINDS and field.name != 'rejected')A corpus’s counts by kind but its rejections, which a position version 7 saved kept alone
(dew.data.plan.earlier_position).
Chunked
Section titled “Chunked”class Chunked( parent: pygrain.IterDataset[Record], name: str, label: str, expand: Callable[[CorpusCounts], gx.FlatMapTransform],)A chunk expansion counted at its fan-out, with counters in the iterator position: its documents
and their chunks, and no row (CorpusCounts.rows), which the counters about it count.
Carried
Section titled “Carried”class Carried(parent: pygrain.IterDataset[Batch | Rejected])parent’s rows, each marker’s source examples (Rejected.consumed) carried on to the next row and
the markers dropped, counting nothing: the carrier after a step whose markers no corpus’s count
follows, a batched map’s group it makes no rows of (dew.data.lowering._Groups). The examples it
owes are checkpointed with its parent’s position; those the stream ends owing are its end’s
(dew.training._fit.drained).
Accepted
Section titled “Accepted”class Accepted( parent: pygrain.IterDataset[Batch | Rejected] | pygrain.IterDataset[Batch], name: str, label: str, *, tokens: Callable[[Batch], int] | None = None, certain_after: int | None = None, rows: bool = True, examples: bool = False, members: Sequence[tuple[str, str, str]] = (), credits: bool = False,)Grain’s filter, with checkpointed counts. A filter of an endless repeat whose passes it
rejects alike refuses a run of certain_after rejections, two whole passes, as reading nothing
forever; any other run of rejections it logs and reads on, a fetch that failed for now say. One
past an objective’s step that makes rows of source examples counts none of the rows it keeps
(rows false, CorpusCounts.rows). A mixture’s counter past the mix counts what it rejects of each
member’s rows under that member (members, each’s name and label), and its last counter, which
credits them, the examples each member’s rows complete as it delivers them (MEMBERS); each
member is its key, name and label.
token_weights
Section titled “token_weights”def token_weights( names: Sequence[str], weights: Sequence[float], windows: Sequence[int], targets: Sequence[int],) -> tuple[float, ...]Window shares for authoritative per-corpus valid-target counts.
Proportions
Section titled “Proportions”class Proportions(proportions: tuple[int, ...], seed: int | None = None)A weighted mix’s selection policy, Grain’s (MapDataset.mix): the weights scaled so the least
is 100 and truncated to whole proportions, and position k of the mix read from the corpus whose
count of records a prefix holds grows from k to k + 1. That schedule repeats every window of the
mix, its proportions’ sum of positions, in which each corpus reads its proportion; a seed
permutes the corpus choices within each window by a permutation of its own (window), so the
mix reads its corpora in no fixed order and every window still holds each at its share. A phase
counts its prefix by it (counts) as the mix reads by it (ProportionalSelection), so the two
never disagree.
Proportions.of
Section titled “Proportions.of”def of(weights: Sequence[float], seed: int | None = None) -> ProportionsProportions.window
Section titled “Proportions.window”def window(index: int) -> tuple[np.ndarray, np.ndarray]The indexth window of the mix: the corpus each of its positions reads, and how many of that
corpus’s records the window read before it. Unseeded, every window is Grain’s schedule; seeded,
window index is it permuted by np.random.default_rng([seed, index]), as a shard order’s
epochs are (dew.data.ordering).
Proportions.counts
Section titled “Proportions.counts”def counts(records: int) -> tuple[int, ...]How many records of each corpus the first records positions of the mix read.
Proportions.covering
Section titled “Proportions.covering”def covering(sizes: Sequence[int]) -> intThe fewest positions of the mix in which every corpus reads sizes of its records, its
size: the pass in which each repeated corpus has come round once.
Proportions.at
Section titled “Proportions.at”def at(position: int) -> tuple[int, int]The corpus position position of the mix reads, the one whose count grows with it, and
which of that corpus’s records: the next it hasn’t read.
Proportions.run
Section titled “Proportions.run”def run(start: int, positions: int) -> tuple[np.ndarray, np.ndarray] | Noneat of positions positions in a row from start, as two arrays, corpus and record:
counts’ split taken for every position at once, a corpus at a time, so a run costs what one
position does in Python and a corpus’s arithmetic in its vector. None where a position’s
products would leave 64-bit integers, which at reads exactly.
ProportionalSelection
Section titled “ProportionalSelection”class ProportionalSelection( lengths: Sequence[int], weights: Sequence[float], seed: int | None = None,)Grain’s weighted mix of corpora of lengths records as a selection map
(MapDataset.select_from_datasets), by Proportions, its corpus choices permuted within each
window by seed where one is given: as long as the mix reads no record of any corpus twice.
RUN-
The positions read ahead together once a reader reads one after another (
Proportions.run).
TokenDeficit
Section titled “TokenDeficit”class TokenDeficit( parents: Sequence[pygrain.IterDataset[Batch]], weights: Sequence[float], keys: Sequence[str], *, tokens: Callable[[Batch], int], names: Sequence[str], labels: Sequence[str],)Pick the corpus furthest below its token share, with index-order ties.
KeyedMix
Section titled “KeyedMix”class KeyedMix( parents: Sequence[pygrain.IterDataset[Batch]], weights: Sequence[float], keys: Sequence[str], seed: int | None = None,)Grain’s weighted mix of corpora, each read as it is drawn (_MixReads), and
each keyed by its member’s key (keys): Grain saves a keyed mix’s positions by key and
restores those it finds, so a member that joins starts at its first row and one that left is
dropped while the others resume where they were. With a seed, the corpus each position reads is
Proportions(seed=...)’s, its windows’ choices permuted (_SeededMix), its position saved alike.
rewound
Section titled “rewound”def rewound( start: Mapping[str, object], current: Mapping[str, object],) -> Mapping[str, object]start, the position a repeat rewinds its pass to, with each counter’s counts as current
holds them (COUNTER: its counts, run of rejections and warning): a pass read again goes on
counting the corpus, where a restored position replaces its counts. Every other part of a
position is the iterator’s own and is left as start has it. A mix’s corpus the pass had not
opened at its start (_MixRead’s _UNOPENED) is rewound to that, carrying the position it holds
now, open or not, whose counters, every one its stages keep, it takes on as it opens again.
state_counts
Section titled “state_counts”def state_counts(state: Mapping[str, object]) -> dict[str, CorpusCounts]Read per-corpus counters through Grain’s nested iterator checkpoints.
Within one iterator a corpus’s counters are its stages’: the outermost counts the rows it delivered and those of them cut short, and each its own rejections. The iterators Grain interleaves, its worker processes or a corpus’s shards, each read records of their own, so their counts add, a shard waiting past the cycle (a future state) included.
data_metrics
Section titled “data_metrics”def data_metrics(counts: Iterable[CorpusCounts]) -> dict[str, float]The metrics a run logs of its corpora (CorpusCounts.metrics), each under its label; corpora
that share one are numbered in the order of their identities, past every label a corpus has.