Skip to content

dew.data.pipeline

Immutable record pipelines, lowered to Grain by :func:compile.

NameSummary
Rejected
CONSUMEDA row’s count of the source examples it is the last row of, set on each source example a run bounded in examples reads (MarkConsumed) and carried through every step to the batch: each example’s count goes to the last row a step makes of it, the next row kept where a step rejects or makes none of it (Owed), and a packed window’s is its rows’ sum.
REFUSEDThe part of a row’s CONSUMED that are source examples a step rejected or made no row of, which the row carries on (Owed): consumed, and none of them trained.
ACCOUNTSA row’s two counts of source examples, which every step carries together, as integers no batch stacks.
MEMBERThe mixture member a row was drawn from, by its key (dew.data.analysis._member_keys), which a member keeps when another joins or leaves, set where a mixture a run bounds in examples draws it (dew.data.lowering._Build.mixed) and carried to every row a step makes of it, its rejection’s marker too (Rejected.member).
MEMBERSThe source examples a row completes, delivered (CONSUMED less REFUSED), by the mixture member each was drawn from (MEMBER), set where a member’s row first holds counts (consumed_by) and carried wherever its counts go, so a mixture counts each member’s examples where they are delivered (dew.data.mixing.CorpusCounts).
COUNTINGA row’s fields the run counts it by, which every step carries out of its sight.
MembersThe source examples a row completes by mixture member (MEMBERS).
member_ofThe mixture member row was drawn from (MEMBER), none for a row of none or of several: every read of a row’s member.
members_ofrow’s completed examples by mixture member (MEMBERS), none where it holds no member’s.
joinedCompleted examples by mixture member, summed member by member, none for a member of none.
DOCUMENTA layout’s field beside its tokens that says, by an id, the document each token is of (Documents), the id negated at the document’s last token, so a window says which documents it holds and which end in it.
DocumentsThe documents a layout lays out of a mixture’s rows, each under the id its tokens carry (DOCUMENT): the member it was drawn from (MEMBER, none for one of several members’ rows) and the examples it completes by member (MEMBERS), until the window that holds its last token takes them.
accountsThe source examples row consumes and the part of them refused (CONSUMED, REFUSED), none where it holds neither: a rejected row’s are all refused, as the row kept after it is paid them (Owed).
MarkConsumedThe step that marks each source example a run bounded in examples reads as one consumed by the row it is read in (CONSUMED), set where the bound is (dew.data.compile).
OwedThe source examples rows a step dropped or made none of were the last of (Rejected.consumed), owed to the next row kept: the one carrier the reading process’s (dew.data.mixing.Accepted), a batched step’s (dew.data.mixing.Carried) and a loading worker’s group (dew.data.lowering) use.
consumed_bystep’s outputs of given, the fields the run counts it by (COUNTING) kept out of the step’s sight: the mixture member it was drawn from (MEMBER) on every output, a rejected row’s marker too (Rejected.member), and the source examples it consumes (CONSUMED, and of them those it carries for refused ones, REFUSED, those it completes by member, MEMBERS) carried to its last output: a row’s counts beside its own, a rejected row’s on its marker (Rejected.consumed, every one of them refused now, completing no member’s), which the corpus’s next kept row carries on (Accepted).
as_memberA step’s output of a mixture member’s row as that member’s (MEMBER, Rejected.member), the examples it completes its member’s where it first holds counts (the bound’s marks).
member_recordA mixture member’s record as the mix reads it, tagged its member’s (as_member).
members_heldThe examples row completes by mixture member (MEMBERS): of its member alone, the consumed it has not refused, where it states no split.
consumed_forwardelements in their order, each rejected row’s consumed examples (Rejected.consumed) carried on to the next row kept after it, as the corpus’s reading carries them (Accepted), so a group that holds its markers apart from its rows (_Grouped) moves none to an earlier row: only a marker no row follows in the group keeps its count, which _Ungrouped hands on after the group’s rows; the row it moves to carries it as refused (REFUSED).
CutsA record’s cut record (_dew_cut): by what each cut, the rows it shortened and the tokens it removed; characters where the kind names them (<kind>_characters), as a cut of text encoded whole is measured.
cutelement cut short by a step that removed removed of it (by what it cut, the tokens), its cut record composed with what earlier steps recorded (composed).
composedTwo cut records of the same rows, one step after another: a kind both cut counts its rows once, the most either shortened, and adds what each removed; a 100-token text cut to 80, then to 60, removed 40 of one row.
cut_unitThe unit a cut of kind removes: characters for a kind that names them (<kind>_characters, a text encoded whole), none for an event that removes nothing (options_alike, a row whose cut left two options alike), else tokens.
cut_countsSeveral rows’ cuts (each the tokens a cut removed of the row, by what it cut) as one record’s _dew_cut, which dew.data.mixing.CorpusCounts.delivered counts: by what each cut, the rows it shortened and the tokens it removed; distinct rows add, where composed adds one row’s successive cuts.
RefusedA request a served task refuses, under the reason its training would reject the same record under (Rejected), which a server counts as a run’s data counters count rejections (dew.data.mixing.CorpusCounts).
Record
RecordStep
RandomStep
HostStep
ColumnStepA function of records read as a dict of columns, each a list, as Hugging Face’s map(batched=True) reads them.
Rejecting
ReasonedA predicate that names what it drops (images.MinSide’s “too_small”).
PACK_POLICYHow a pack fills its rows where none is asked for.
DROP_REMAINDERWhether a batch a training pass fills only in part is dropped, where none is asked for.
TOKEN_FIELDSThe per-token fields a pack or a window lays out where none are named.
PACK_BINSThe rows a pack holds open at once where none is asked for.
INTERLEAVEThe sources an interleave reads at once where none is asked for.
PREFETCHThe records a prefetch reads ahead where none is asked for.
Volatile
ExternalA step that reads the storage a record names by path, as a media decoder does when its field holds a path rather than bytes: what it makes is keyed by the path, not by what the path holds, so no automatic cache’s prefix runs past it (dew.data.compile._automatic), a file replaced at the same path reading anew; a chain’s own cache= keeps what it made, as it is told to.
CostlyA step that costs more to run on a record than reading what it made back from a cache costs, as measured (tools/benchmark_data.py automatic-cache), and makes it from the record alone: so a chain that runs it first caches what it makes with no cache= (dew.data.compile._automatic).
SingleReader
TokenGeometryA record step, or a stage that lays rows out (Pack, Windows), whose rows each hold max_length tokens, which a token budget divides into rows (TokenBudget).
TokenBudgetA batch of tokens tokens: as many rows as the pipeline’s fixed row length (TokenGeometry) divides it into, resolved when it compiles.
Expansion
ExpandStep
DeclaresA step, or a stage, that says what it makes of the facts of the records it reads (Dataset.facts): the fields it writes and their shapes, the tokenizer it encodes with.
StatedA store that states facts of its records without reading them (Dataset.facts): a token store’s tokenizer and vocabulary, say.
MeasureA fact a configured step needs that a memory read measures over a store’s records where the store states none (Configured.measures): fact names it, and calling it on the records gives its value.
ConfiguredResolve one record step from source facts before any record reader opens.
BoundThe configured step a record step was resolved from, with the metadata values its factory read.
BatchedA function of size records at a time, read as a dict of columns, each a list, as Hugging Face’s map(batched=True): with same_rows, its columns of as many rows join the records’ own; otherwise they are the records that replace the group.
NativeA function of the Grain dataset the stages before it make, for a Grain operation Dew does not name; the rows after it are read per share.
Map
examples_boundWhere a run bounded in examples marks the source examples it reads (MarkConsumed), the stage after the bound’s limit (dew.data.compile._bounded): its index in stages, None for a read the run bounds otherwise.
RandomMap
MapWithIndex
Filter
FlatMap
Shuffle
Repeat
Slice
Interleave
BatchStage
PackDocuments packed into windows of seq_len tokens along fields.
Windows
TokenSplit
Prefetch
Workers
Limit
DeviceStepA step over the training batch that runs inside the compiled training step, on the accelerator, rather than in the reader: pure and traceable, a batch of arrays in and out, its randomness only from per-record key fields (images.AugmentationKey), so its draws are the same at any split, microbatch or pool size and on resume.
MapBatch
CachedWhat a cache stage fills and reads, a dew.data.cache.Cache, or the published generation it stands in for, a dew.data.generation.Generation: each says where it lives and which rows a plan reads from it.
Artifact
Stage
StepWhat a stage runs and its identity names (dew.data.plan.StepIdentity): a record, random, indexed, expanding, batch or column step, a predicate, a Grain function or a configured step’s factory.
RunsWhat a stage runs (runs): its callable, the identity written for it, and the configured step it was bound from, the one description its plan identity, its cache keyability and show read.
runsWhat stage runs, None for a stage that runs no step of its own (an order, a batch, a cache).
Mixture
Phases
childrenThe chains pipeline’s mixture or phases read, none where it reads a store.
chainsEvery chain of pipeline’s tree, children first: each corpus under its mixtures and phases, then the chain that mixes or phases them, and pipeline last.
with_childrenpipeline with each chain its mixture or phases read rewritten by rewrite; pipeline itself where it reads a store.
with_storespipeline with each store under its mixtures and phases read through store, every step as it was.
DatasetRecords a run reads: a store (Files, Hub, TFDS, Memory, Torch), or a store with the steps the verbs added, in the order they are written.
PipedA call of Dataset.pipe and what it made: function(input, *args, **settings) made the stages start to end of the chain that keeps it, or, where it made another store of its input (a token store’s windows), the store and the stages to end from input.
PipelineA dataset’s store and the stages its verbs added; what every verb returns.
concatThe records of datasets, stores read one after another, as one dataset.
mixdatasets read together, each filling its share of a step, as Grain’s mix: weights are the shares, equal by default, counted in records, or with by="tokens" in the valid target tokens of each member’s field.
phasesDatasets read one after another: each (dataset, until_step) until the training step it names, the last until the run ends (None).
scheduleDatasets read one after another on the run’s committed-token clock, the clock fit(tokens=) counts: each (dataset, tokens) for the valid target tokens its steps commit, the last for the rest of the run (None), or for its tokens, where the data ends (the run’s last step).
ObjectiveData
record_stageThe stage a record step is applied by where no verb names it, from what the step says it is: an expansion’s records (FlatMap), a generator’s draws (RandomMap), or one record’s replacement (Map), dropping records where it says it does (rejects).

dataclass source

class Rejected(
reason: str,
permanent: bool = True,
consumed: int = 0,
cut: Mapping[str, int] = dict(),
member: str | None = None,
)
consumed: int

The source examples a rejected row was the last of (CONSUMED), which the next row its corpus keeps carries on (dew.data.mixing.Accepted).

cut: Mapping[str, int]

The tokens the cuts of the refused rows removed, by what each cut, which its corpus counts beside the refusal (dew.data.mixing.CorpusCounts.dropped).

member: str | None

The mixture member the rejected row was drawn from (MEMBER), which its refusal is counted under.

attribute source

CONSUMED = '_dew_consumed'

A row’s count of the source examples it is the last row of, set on each source example a run bounded in examples reads (MarkConsumed) and carried through every step to the batch: each example’s count goes to the last row a step makes of it, the next row kept where a step rejects or makes none of it (Owed), and a packed window’s is its rows’ sum. A batch’s sum is the source examples its dispatch consumes (dew.batch.EXAMPLES_CONSUMED).

attribute source

REFUSED = '_dew_refused'

The part of a row’s CONSUMED that are source examples a step rejected or made no row of, which the row carries on (Owed): consumed, and none of them trained. It travels with CONSUMED through every step, so a count of the examples delivered rows complete takes their difference (dew.data.mixing.CorpusCounts).

attribute source

ACCOUNTS = (CONSUMED, REFUSED)

A row’s two counts of source examples, which every step carries together, as integers no batch stacks.

attribute source

MEMBER = '_dew_member'

The mixture member a row was drawn from, by its key (dew.data.analysis._member_keys), which a member keeps when another joins or leaves, set where a mixture a run bounds in examples draws it (dew.data.lowering._Build.mixed) and carried to every row a step makes of it, its rejection’s marker too (Rejected.member).

attribute source

MEMBERS = '_dew_members'

The source examples a row completes, delivered (CONSUMED less REFUSED), by the mixture member each was drawn from (MEMBER), set where a member’s row first holds counts (consumed_by) and carried wherever its counts go, so a mixture counts each member’s examples where they are delivered (dew.data.mixing.CorpusCounts). Examples a step refused are carried on as refused (Owed) and complete no member’s: what is refused of a member is counted where it is refused (Rejected.member).

attribute source

COUNTING = (*ACCOUNTS, MEMBER, MEMBERS)

A row’s fields the run counts it by, which every step carries out of its sight.

attribute source

The source examples a row completes by mixture member (MEMBERS).

function source

def member_of(row: Batch) -> str | None

The mixture member row was drawn from (MEMBER), none for a row of none or of several: every read of a row’s member.

function source

def members_of(row: Batch) -> Members

row’s completed examples by mixture member (MEMBERS), none where it holds no member’s.

function source

def joined(*members: Members = ()) -> dict[str, int]

Completed examples by mixture member, summed member by member, none for a member of none.

attribute source

DOCUMENT = '_dew_document'

A layout’s field beside its tokens that says, by an id, the document each token is of (Documents), the id negated at the document’s last token, so a window says which documents it holds and which end in it.

dataclass source

class Documents(
next: int = 1,
held: dict[int, tuple[str | None, dict[str, int]]] = dict(),
)

The documents a layout lays out of a mixture’s rows, each under the id its tokens carry (DOCUMENT): the member it was drawn from (MEMBER, none for one of several members’ rows) and the examples it completes by member (MEMBERS), until the window that holds its last token takes them.

def laid(member: str | None, members: Members) -> int

The id of a document of member that completes members.

def saved(documents: Iterable[int]) -> dict[str, JSON]

documents’ members and examples as a saved position holds them, by id.

def restore(saved: JSON) -> None

Hold the documents a saved position held (saved) again.

function source

def accounts(row: Batch | Rejected) -> tuple[int, int]

The source examples row consumes and the part of them refused (CONSUMED, REFUSED), none where it holds neither: a rejected row’s are all refused, as the row kept after it is paid them (Owed).

dataclass source

class MarkConsumed()

The step that marks each source example a run bounded in examples reads as one consumed by the row it is read in (CONSUMED), set where the bound is (dew.data.compile).

dataclass source

class Owed(examples: int = 0)

The source examples rows a step dropped or made none of were the last of (Rejected.consumed), owed to the next row kept: the one carrier the reading process’s (dew.data.mixing.Accepted), a batched step’s (dew.data.mixing.Carried) and a loading worker’s group (dew.data.lowering) use.

def owe(consumed: int) -> None

Owe consumed more examples to the next row kept.

def paid(row: Batch) -> Batch

row, kept, taking the examples owed (CONSUMED), every one refused (REFUSED), none owed after it.

function source

def consumed_by(
given: Record,
step: Callable[[Record], tuple[Record, ...]],
) -> tuple[Record, ...]

step’s outputs of given, the fields the run counts it by (COUNTING) kept out of the step’s sight: the mixture member it was drawn from (MEMBER) on every output, a rejected row’s marker too (Rejected.member), and the source examples it consumes (CONSUMED, and of them those it carries for refused ones, REFUSED, those it completes by member, MEMBERS) carried to its last output: a row’s counts beside its own, a rejected row’s on its marker (Rejected.consumed, every one of them refused now, completing no member’s), which the corpus’s next kept row carries on (Accepted). Every other row holds none, so the rows of one read hold the same fields. A member’s row that first holds counts (the bound’s marks, MarkConsumed) completes them as its member’s.

function source

def as_member(made: Record, member: str | None) -> Record

A step’s output of a mixture member’s row as that member’s (MEMBER, Rejected.member), the examples it completes its member’s where it first holds counts (the bound’s marks). A record of raw bytes holds no field to say it and passes as it is: the mixture refuses a member’s bytes where it tags its records (member_record), and a step after the bound that makes bytes of a row is refused where its counts are carried (consumed_by).

function source

def member_record(made: Record, member: str) -> Record

A mixture member’s record as the mix reads it, tagged its member’s (as_member). A member’s raw bytes hold no field to say it, so its examples, which a mixture counting its members’ would credit to none, are refused, not passed on untagged: a mixture’s members are decoded before the mix (row 3746).

function source

def members_held(row: Batch, member: str | None) -> dict[str, int]

The examples row completes by mixture member (MEMBERS): of its member alone, the consumed it has not refused, where it states no split.

function source

def consumed_forward(elements: Sequence[Record]) -> list[Record]

elements in their order, each rejected row’s consumed examples (Rejected.consumed) carried on to the next row kept after it, as the corpus’s reading carries them (Accepted), so a group that holds its markers apart from its rows (_Grouped) moves none to an earlier row: only a marker no row follows in the group keeps its count, which _Ungrouped hands on after the group’s rows; the row it moves to carries it as refused (REFUSED).

attribute source

A record’s cut record (_dew_cut): by what each cut, the rows it shortened and the tokens it removed; characters where the kind names them (<kind>_characters), as a cut of text encoded whole is measured.

function source

def cut(element: Batch, removed: Mapping[str, int]) -> Batch

element cut short by a step that removed removed of it (by what it cut, the tokens), its cut record composed with what earlier steps recorded (composed).

function source

def composed(earlier: Cuts, later: Cuts) -> dict[str, tuple[int, int]]

Two cut records of the same rows, one step after another: a kind both cut counts its rows once, the most either shortened, and adds what each removed; a 100-token text cut to 80, then to 60, removed 40 of one row.

function source

def cut_unit(kind: str) -> str | None

The unit a cut of kind removes: characters for a kind that names them (<kind>_characters, a text encoded whole), none for an event that removes nothing (options_alike, a row whose cut left two options alike), else tokens.

function source

def cut_counts(cuts: Iterable[Mapping[str, int]]) -> dict[str, tuple[int, int]]

Several rows’ cuts (each the tokens a cut removed of the row, by what it cut) as one record’s _dew_cut, which dew.data.mixing.CorpusCounts.delivered counts: by what each cut, the rows it shortened and the tokens it removed; distinct rows add, where composed adds one row’s successive cuts.

class source

class Refused(reason: str, message: str)

A request a served task refuses, under the reason its training would reject the same record under (Rejected), which a server counts as a run’s data counters count rejections (dew.data.mixing.CorpusCounts).

attribute source

attribute source

attribute source

attribute source

attribute source

A function of records read as a dict of columns, each a list, as Hugging Face’s map(batched=True) reads them.

class source

class Rejecting(Protocol)

class source

class Reasoned(Protocol)

A predicate that names what it drops (images.MinSide’s “too_small”).

attribute source

PACK_POLICY: Literal['first-fit', 'best-fit', 'concat'] = 'first-fit'

How a pack fills its rows where none is asked for.

attribute source

DROP_REMAINDER = True

Whether a batch a training pass fills only in part is dropped, where none is asked for. Grain keeps it (drop_remainder=False); Dew drops it because the training step is compiled for one batch shape, and a short batch would compile a second program and take an update over fewer rows. A validation pass never drops one (dew.data.lowering._Build.batch_order), and a run bounded in examples keeps its last batch, filled and marked (dew.data.compile._bounded).

attribute source

TOKEN_FIELDS = ('text',)

The per-token fields a pack or a window lays out where none are named.

attribute source

PACK_BINS = 32

The rows a pack holds open at once where none is asked for.

attribute source

INTERLEAVE = 16

The sources an interleave reads at once where none is asked for.

attribute source

PREFETCH = 64

The records a prefetch reads ahead where none is asked for.

class source

class Volatile(Protocol)

class source

class External(Protocol)

A step that reads the storage a record names by path, as a media decoder does when its field holds a path rather than bytes: what it makes is keyed by the path, not by what the path holds, so no automatic cache’s prefix runs past it (dew.data.compile._automatic), a file replaced at the same path reading anew; a chain’s own cache= keeps what it made, as it is told to.

class source

class Costly(Protocol)

A step that costs more to run on a record than reading what it made back from a cache costs, as measured (tools/benchmark_data.py automatic-cache), and makes it from the record alone: so a chain that runs it first caches what it makes with no cache= (dew.data.compile._automatic). An image decode is not one: an image a record names by path is read from storage the record’s content does not key.

class source

class SingleReader(Protocol)

class source

class TokenGeometry(Protocol)

A record step, or a stage that lays rows out (Pack, Windows), whose rows each hold max_length tokens, which a token budget divides into rows (TokenBudget).

dataclass source

class TokenBudget(
tokens: int,
drop_remainder: bool = DROP_REMAINDER,
batch_fn: Callable[[Sequence[Batch]], Batch] | None = None,
)

A batch of tokens tokens: as many rows as the pipeline’s fixed row length (TokenGeometry) divides it into, resolved when it compiles.

def declared(facts: Mapping[str, object]) -> Mapping[str, object]

Rows of the pipeline’s row length, as many a batch as tokens holds.

class source

class Expansion(Protocol)
def flat_map(element: Batch, /) -> Sequence[Batch | Rejected]

attribute source

class source

class Declares(Protocol)

A step, or a stage, that says what it makes of the facts of the records it reads (Dataset.facts): the fields it writes and their shapes, the tokenizer it encodes with.

def declared(facts: Mapping[str, object], /) -> Mapping[str, object]

class source

class Stated(Protocol)

A store that states facts of its records without reading them (Dataset.facts): a token store’s tokenizer and vocabulary, say.

def stated() -> Mapping[str, object]

class source

class Measure(Protocol)

A fact a configured step needs that a memory read measures over a store’s records where the store states none (Configured.measures): fact names it, and calling it on the records gives its value.

dataclass source

class Configured(
factory: Callable[[Mapping[str, object]], RecordStep | pygrain.RandomMapTransform | Expansion],
needs: tuple[str, ...],
scope: Literal['source', 'run'] = 'source',
random: bool = False,
one_to_one: bool = False,
measures: Measure | None = None,
expands: bool = False,
examples: bool = False,
)

Resolve one record step from source facts before any record reader opens. one_to_one states that the step keeps one record for each input whatever the facts, which a cache after it reads before it binds; binding it to a step that drops or expands records is refused. measures measures a fact it needs over each store it reads, in a memory read: the whole store, below any holdout, so the parts a run trains and scores on agree; a store’s own statement of it gives way to the measure, and a mixture’s step binds the largest of its stores’. random and expands state what the step it binds to is, before it binds: a random map’s, and an expansion’s of a record into several (Expansion); binding to another is refused. An objective’s step that expands is marked as one that makes several rows of each source example (examples, set where the objective places it): it runs after the chain’s record order, a run’s examples= bound stands before it, and the expansion it binds to carries the mark (FlatMap.examples).

dataclass source

class Bound(step: Configured, values: Mapping[str, object])

The configured step a record step was resolved from, with the metadata values its factory read.

dataclass source

class Batched(
step: ColumnStep,
size: int,
identity: str | None = None,
same_rows: bool = True,
)

A function of size records at a time, read as a dict of columns, each a list, as Hugging Face’s map(batched=True): with same_rows, its columns of as many rows join the records’ own; otherwise they are the records that replace the group. The last group of a finite pass may be smaller.

dataclass source

class Native(
function: Callable[[pygrain.MapDataset | pygrain.IterDataset], pygrain.MapDataset | pygrain.IterDataset],
identity: str | None = None,
)

A function of the Grain dataset the stages before it make, for a Grain operation Dew does not name; the rows after it are read per share.

dataclass source

class Map(
step: RecordStep,
identity: str | None = None,
rejects: bool = False,
configured: Bound | None = None,
)

function source

def examples_bound(stages: Sequence[object]) -> int | None

Where a run bounded in examples marks the source examples it reads (MarkConsumed), the stage after the bound’s limit (dew.data.compile._bounded): its index in stages, None for a read the run bounds otherwise. The one locator of the bound every compile and lowering step asks.

dataclass source

class RandomMap(
step: RandomStep,
identity: str | None = None,
rejects: bool = False,
configured: Bound | None = None,
)

dataclass source

class MapWithIndex(
step: Callable[[int, Batch], Batch | Rejected],
identity: str | None = None,
rejects: bool = False,
)

dataclass source

class Filter(
predicate: Callable[[Batch], bool],
reason: str = 'filtered',
identity: str | None = None,
)

dataclass source

class FlatMap(
step: ExpandStep,
identity: str | None = None,
configured: Bound | None = None,
examples: bool = False,
)
examples: bool

Whether this is an objective’s step that makes several rows of each source example (a decision layout’s row a question): a corpus counts its source examples before it (Accepted), and a run’s examples= bound stands before it (dew.data.compile._bounded).

dataclass source

class Shuffle(
seed: int,
window: int | None = None,
block_shards: int | None = None,
streamed: int | None = None,
)
streamed: int | None

Where window is None, the records a streamed read draws its order from, whose records no index permutes whole (Pipeline.shuffle); None for a shuffle a streamed read refuses.

def streaming() -> Shuffle

This shuffle as a read that streams takes it: within its streamed window where it states one, as a window shuffle, so its loading workers hand on where it stands; itself otherwise. Every lowering of a streamed read resolves it here (dew.data.lowering).

dataclass source

class Repeat(epochs: int | None = None)

dataclass source

class Slice(selection: slice)

dataclass source

class Interleave(cycle_length: int = INTERLEAVE)

dataclass source

class BatchStage(
size: int,
drop_remainder: bool = DROP_REMAINDER,
batch_fn: Callable[[Sequence[Batch]], Batch] | None = None,
)
def declared(facts: Mapping[str, object]) -> Mapping[str, object]

The same records, size of them a batch.

dataclass source

class Pack(
seq_len: int,
fields: tuple[str, ...] = TOKEN_FIELDS,
bins: int = PACK_BINS,
policy: Literal['first-fit', 'best-fit', 'concat'] = PACK_POLICY,
segments: int | None = None,
along: tuple[tuple[str, int], ...] = (),
overlap: int = 0,
)

Documents packed into windows of seq_len tokens along fields.

along packs other fields on axes of their own, each to its own length, and segments bounds the documents a window holds. A first-fit or best-fit window closes when any of them is full, as Grain’s packer does with its length_struct and max_sequences_per_bin.

A concatenating pack cuts documents where its windows end; overlap is how many tokens before a cut through a document the next window holds again, so 1 keeps the next-token target at every cut. A fitting pack cuts no document: one longer than a window is cut before it, into pieces that overlap as dew.data.text.Chunks says.

packed: tuple[str, ...]

Every field a window holds: fields and those along axes of their own.

max_length: int

The inputs each window holds, its last token their target, which a token budget divides into rows (TokenGeometry).

def document(element: Batch) -> Batch

element as a document a window packs: its packed fields alone, its token fields of one length (dew.data.steps.generic.aligned), its along fields beside them.

def declared(facts: Mapping[str, object]) -> Mapping[str, object]

Rows of seq_len tokens along fields, the last of them a target: seq_len - 1 inputs.

dataclass source

class Windows(
seq_len: int,
stride: int | None = None,
fields: tuple[str, ...] = TOKEN_FIELDS,
)
max_length: int

The inputs each window holds, which a token budget divides into rows (TokenGeometry).

def declared(facts: Mapping[str, object]) -> Mapping[str, object]

Windows of seq_len inputs and their targets along fields.

dataclass source

class TokenSplit(fraction: float, held: bool, fields: tuple[str, ...] = TOKEN_FIELDS)

dataclass source

class Prefetch(buffer: int = PREFETCH)

dataclass source

class Workers(count: int, buffer: int = WORKER_BUFFER)

dataclass source

class Limit(count: int, passes: int | None = None)
passes: int | None

Where set, the limit is that many passes over the rows reaching it, which a memory read that holds them counts as it holds them and sets count to (dew.data.materialize): a run of epochs’s bound (dew.data.compile._bounded), the stages the read ran stating the count it resolved to.

class source

class DeviceStep

A step over the training batch that runs inside the compiled training step, on the accelerator, rather than in the reader: pure and traceable, a batch of arrays in and out, its randomness only from per-record key fields (images.AugmentationKey), so its draws are the same at any split, microbatch or pool size and on resume. The reader leaves the fields it reads in the batch it delivers (Loader.device).

dataclass source

class MapBatch(step: HostStep | DeviceStep, identity: str | None = None)

class source

class Cached(Protocol)

What a cache stage fills and reads, a dew.data.cache.Cache, or the published generation it stands in for, a dew.data.generation.Generation: each says where it lives and which rows a plan reads from it.

def reference(manifest: DependencyManifest, /) -> str

dataclass source

class Artifact(cache: Cached, automatic: bool = False)
automatic: bool

Whether the compiler placed it (dew.data.compile._automatic) rather than the chain: it holds the records its prefix makes, so a run reads the same records with or without it and its plan names none (dew.data.cache.automatic_root).

attribute source

attribute source

What a stage runs and its identity names (dew.data.plan.StepIdentity): a record, random, indexed, expanding, batch or column step, a predicate, a Grain function or a configured step’s factory.

dataclass source

class Runs(step: Step, identity: str | None = None, configured: Bound | None = None)

What a stage runs (runs): its callable, the identity written for it, and the configured step it was bound from, the one description its plan identity, its cache keyability and show read.

function source

def runs(stage: Stage) -> Runs | None

What stage runs, None for a stage that runs no step of its own (an order, a batch, a cache).

dataclass source

class Mixture(
pipelines: tuple[Pipeline, ...],
weights: tuple[float, ...],
tokens: Callable[[Batch], int] | None = None,
seed: int | None = None,
)
seed: int | None

The seed whose permutation of each window’s corpus choices the mix reads in (Proportions): None reads Grain’s schedule, the same every window in the members’ order.

dataclass source

class Phases(
pipelines: tuple[Pipeline, ...],
ends: tuple[int | None, ...],
clock: Literal['steps', 'tokens'] = 'steps',
)
ends: tuple[int | None, ...]

Where each phase ends, as clock counts them: at the training step it names (phases), or at the run’s committed tokens (schedule). The last runs on (None); on the token clock it may end instead, where the data ends.

function source

def children(pipeline: Pipeline) -> tuple[Pipeline, ...]

The chains pipeline’s mixture or phases read, none where it reads a store.

function source

def chains(pipeline: Pipeline) -> Iterator[Pipeline]

Every chain of pipeline’s tree, children first: each corpus under its mixtures and phases, then the chain that mixes or phases them, and pipeline last.

function source

def with_children(
pipeline: Pipeline,
rewrite: Callable[[Pipeline], Pipeline],
) -> Pipeline

pipeline with each chain its mixture or phases read rewritten by rewrite; pipeline itself where it reads a store.

function source

def with_stores(pipeline: Pipeline, store: Callable[[Source], Source]) -> Pipeline

pipeline with each store under its mixtures and phases read through store, every step as it was.

class source

class Dataset

Records a run reads: a store (Files, Hub, TFDS, Memory, Torch), or a store with the steps the verbs added, in the order they are written. A dataset never changes; every verb returns a new one. Trainer.fit and iterate lower it to Grain.

The verb shows the cardinality: map is one record to one, random_map one to one with a generator, filter one to none or one, flat_map one to many; dataset-level stages, such as text.pack, are functions of a dataset applied with pipe. The compiler takes each stage where it stands or refuses it before a reader opens.

batched: bool

Whether this chain has made its training batch, after which map reads batches.

def holdout(
fraction: float | None = None,
*,
key: str | None = None,
records: int | None = None,
split: str | tuple[str, ...] | None = None,
tokens: float | None = None,
fields: tuple[str, ...] = TOKEN_FIELDS,
**named: float = {},
) -> tuple[Dataset, ...]

This dataset split into the records a run trains on and the ones it scores, with the steps it has replayed on each, so hold out before preparing.

holdout(0.01) returns (train, held), and holdout(val=0.0005, test=0.0005) the remainder and one dataset per name, in order: a record goes to the part its identity hashes into, the same in every run, on every machine and after every resume. key names the field that identifies a document (an id or a url), so its versions and chunks land together; without one, a record’s identity is its whole content. records=n holds out the first n records, and split="validation" the store’s own split, or split=("validation", "test") each of them, in order, after the records the store reads by default. tokens=0.01 holds out the first hundredth of the tokens of the order the steps so far read, cutting at most one document in two, so the parts are disjoint and together the corpus; it reads a finite indexed order, before any repeat, and cuts the per-token fields together, counting the first.

def facts() -> Mapping[str, object]

What this dataset’s records are, read without opening a reader: the facts its stores state (a token store’s tokenizer and vocabulary), then what each step makes of them in order: fields, each field’s shape a record holds; seq_len and packed for token rows; tokenizer, vocab_size; batch once batched. A step that changes records and doesn’t say how leaves their fields unknown.

def named(name: str) -> Pipeline

This dataset under name, which its counts log under in a mix.

def pipe(
stages: Callable[Concatenate[Dataset, Options], Dataset],
*args: Options.args = (),
**kwargs: Options.kwargs = {},
) -> Dataset

stages(self, *args, **kwargs): a dataset-level stage, such as text.pack. The chain it returns keeps the call beside the stages it made (Piped), which a run’s record writes in their place (dew.data.records).

def map(
*steps: RecordStep | Configured | HostStep = (),
batched: bool = False,
batch_size: int = 1000,
identity: str | None = None,
) -> Pipeline

One record to one: each step returns its record’s replacement. A step that drops records (returns Rejected) goes through filter_map.

With batched, a step reads batch_size records at a time as a dict of columns, each a list, and returns columns of as many rows, which replace or join the record’s own (Hugging Face’s map(batched=True)). After batch, a step reads each training batch on the host.

def random_map(*steps: RandomStep = (), identity: str | None = None) -> Pipeline

One record to one, each step drawing from the record’s generator (fn(record, rng) or a grain.RandomMapTransform). A draw is keyed by the record’s place in the run’s order, its epoch included: the same in every read mode and reader for one order, and on resume; a different shuffle draws differently, and a read cut per share (one that drops records, or a streamed one) keys it by the record’s place in its share.

def filter_map(*steps: RecordStep = (), identity: str | None = None) -> Pipeline

One record to none or one: each step returns its record’s replacement or Rejected(reason), which drops it, counted by reason in the reader’s checkpointed counts, as images.fetch and images.DecodeImage do.

def random_filter_map(*steps: RandomStep = (), identity: str | None = None) -> Pipeline

One record to none or one, each step drawing from the record’s generator (fn(record, rng) or a grain.RandomMapTransform) and returning its record’s replacement or Rejected(reason), which drops it, counted by reason, as video.PickClip drops a clip shorter than its window. Its draws are keyed as random_map’s are, by the record’s place in the run’s order.

def map_with_index(
step: Callable[[int, Batch], Batch | Rejected],
*,
identity: str | None = None,
) -> Pipeline

Map each row with its position in its source, as the read numbers it.

That is the index of an indexed source, the rows of the earlier shards plus the row’s place in its own for a streamed source, and a cache’s input index. A filter keeps it, so a rejected row leaves a gap, a flat_map’s outputs share their input’s, and shuffles, repeats and slices carry it with the row, so it is the same in every read, under a cache, on resume and at any share or worker count. A row built from several (after pack, windows, group or batch) or a mixture of corpora has no single position, so the step is refused there. A step that drops rows says so with rejects = True and returns Rejected(reason).

def filter(
predicate: Callable[[Batch], bool],
*,
reason: str | None = None,
identity: str | None = None,
) -> Pipeline

Keep the records predicate accepts; the rest are counted under reason, by default the predicate’s own (images.MinSide’s “too_small”).

def flat_map(
step: ExpandStep | ColumnStep,
*,
batched: bool = False,
batch_size: int = 1000,
identity: str | None = None,
) -> Pipeline

One record to any number: the step returns a record’s records. With batched, it reads batch_size records at a time as a dict of columns and returns columns of any length, the records that replace them.

def shuffle(
seed: int = 0,
*,
window: int | None = None,
block_shards: int | None = None,
streamed: int | None = None,
) -> Pipeline

The records in an order drawn from seed: whole, within a window of records, or by blocks of block_shards physical shards; or whole where the read is indexed and within streamed records where it streams, as the read the run takes decides. The seed is 0 where none is given, where Grain derives an unseeded shuffle from its pipeline’s seed (ds.seed), in Dew the run’s (compile(seed=)): this stage’s order is the chain’s own, the one every run of the chain reads and its saved plan names, so a run’s seed changes neither it nor what a resume reads by it. What the run’s seed does draw are its random steps and the order a streamed read deals its physical shards in (dew.data.shards.streamed).

def repeat(epochs: int | None = None) -> Pipeline
def interleave(cycle_length: int = INTERLEAVE) -> Pipeline
def batch(
size: int | None = None,
*,
tokens: int | None = None,
drop_remainder: bool = DROP_REMAINDER,
batch_fn: Callable[[Sequence[Batch]], Batch] | None = None,
) -> Pipeline
def pack(
seq_len: int,
*,
fields: tuple[str, ...] = TOKEN_FIELDS,
bins: int = PACK_BINS,
policy: Literal['first-fit', 'best-fit', 'concat'] = PACK_POLICY,
segments: int | None = None,
along: tuple[tuple[str, int], ...] = (),
overlap: int = 0,
) -> Pipeline

Records laid end to end in rows of seq_len tokens: each row holds the per-token fields of the records it packs, each along field on an axis of its own, and at most segments records; a field that is one value a record has no place in a row of several and is left out. A concatenating pack’s row starts again overlap tokens before a cut through a record (Pack).

def windows(
seq_len: int,
*,
stride: int | None = None,
fields: tuple[str, ...] = TOKEN_FIELDS,
) -> Pipeline
def prefetch(buffer: int = PREFETCH) -> Pipeline
def workers(count: int, *, buffer: int = WORKER_BUFFER) -> Pipeline
def take(count: int) -> Pipeline
def cache(cache: str | Cached) -> Pipeline

Fill cache with the deterministic record steps before it, or read a published Generation as the source in their place, with its published packing in place of a pack that follows.

def snapshot(
url: str,
*,
format: Literal['array_record', 'parquet'] | Writer = 'array_record',
rows_per_shard: int = 100000,
deduplicate: NearDuplicates | None = None,
) -> Mapping[str, object]

Run this dataset once over every record and seal what it makes at url: immutable shards and their committed manifest, which Files(url) reads back, written as ArrayRecord, as Parquet or by a native writer (dew.data.sinks.Writer). A fetch’s outcomes are sealed with it, so a run that reads the snapshot resumes exactly. With deduplicate (dew.data.text.NearDuplicates, the curation extra), each group of near-duplicates in that one pass is sealed as one record, and the manifest’s metadata records the rule, DataTrove’s release and the records removed. Returns the manifest.

def rename_columns(names: Mapping[str, str]) -> Pipeline

Each record with its fields renamed, as rename_columns({"txt": "caption"}).

def select_columns(*names: str = ()) -> Pipeline

Each record with the fields names alone.

def remove_columns(*names: str = ()) -> Pipeline

Each record without the fields names.

def grain(
function: Callable[[pygrain.MapDataset | pygrain.IterDataset], pygrain.MapDataset | pygrain.IterDataset],
*,
identity: str | None = None,
) -> Pipeline

function of the Grain dataset this chain makes, for a Grain operation Dew does not name (zip, Grain’s own interleave, FlatMapMapDataset). The chain after it is read per share and checkpoints through Grain’s own state; identity names it in a run’s plan, its code and closure by default.

def stats(records: int = 1000) -> dict[str, float]

The counts a run logs from a fresh, lazy pass over at most records accepted records: the rows delivered and the drops by reason (dew.data.mixing.CorpusCounts), the rows the pass read a second once its reader was open (data/records_per_second: the loading settings default to none, while workers or prefetching the chain writes run as written), and for token rows, under “text”, the tokens they hold (data/tokens) and the share of their places that pad a row (data/padding).

def read(mode: Read) -> Pipeline

Read the stores this way, rather than as they prefer: “memory” holds every record and runs the deterministic steps once, “indexed” reads by index, “streamed” reads shards in order.

def threads(count: int) -> Pipeline

Read with count threads a worker, rather than the measured default.

def elastic() -> Pipeline

Read it for training so a run resumes on any number of readers that splits its batch, as a global order does, where its own order is read per share (a stream, or steps that drop, expand, pack or draw). Every reader reads the order one reader reads, through its last step that is not a map of each record alone, and keeps every count-th row of each batch, its share; the maps written after that step run on its own rows alone, on its read threads. So its rows are a one-reader run’s, and its position, the same on every reader, resumes on any count, as HF’s split_dataset_by_node reads a stream on every node; each reader reads the steps before those maps over every record. Its loading workers read its inputs in their order, by index or each reading the whole stream and keeping its groups (dew.data.elastic_order), so their count is no part of its position either.

def loading(loading: Loading) -> Pipeline

Read with these workers, threads and buffers (Loading), rather than the measured defaults, recorded with the chain; threads sets its read threads alone.

def show(count: int = 8) -> None

Print the chain, each verb and step a preset or function expanded to in order, then its first count records.

dataclass source

class Piped(
function: Callable[..., Dataset],
args: tuple[object, ...],
settings: tuple[tuple[str, object], ...],
start: int,
end: int,
input: Pipeline | None = None,
)

A call of Dataset.pipe and what it made: function(input, *args, **settings) made the stages start to end of the chain that keeps it, or, where it made another store of its input (a token store’s windows), the store and the stages to end from input.

dataclass source

class Pipeline(
source: Source | Mixture | Phases,
stages: tuple[Stage, ...] = (),
name: str | None = None,
mode: Read | None = None,
_loading: Loading | None = None,
_elastic: bool = False,
piped: tuple[Piped, ...] = (),
)

A dataset’s store and the stages its verbs added; what every verb returns.

piped: tuple[Piped, ...]

The calls of pipe that made its stages (Piped), outermost last; no plan reads them.

function source

def concat(datasets: Sequence[Dataset]) -> Pipeline

The records of datasets, stores read one after another, as one dataset.

function source

def mix(
datasets: Sequence[Dataset],
weights: Sequence[float] | None = None,
*,
by: Literal['records', 'tokens'] = 'records',
field: str = 'text',
seed: int | None = None,
) -> Pipeline

datasets read together, each filling its share of a step, as Grain’s mix: weights are the shares, equal by default, counted in records, or with by="tokens" in the valid target tokens of each member’s field. Each member logs under its name, its store’s by default (Dataset.named). Counted in records, the mix reads its members in Grain’s schedule, which every window of the mix (its proportions’ sum of positions) repeats in the members’ order; a seed permutes each window’s choices by a permutation of its own, every window still holding each member at its share (dew.data.mixing.Proportions).

function source

def phases(*entries: tuple[Dataset, int | None] = ()) -> Pipeline

Datasets read one after another: each (dataset, until_step) until the training step it names, the last until the run ends (None). Each phase is a whole chain, laid out and batched as it reads, each of the same batch.

function source

def schedule(stages: Sequence[tuple[Dataset, float | None]]) -> Pipeline

Datasets read one after another on the run’s committed-token clock, the clock fit(tokens=) counts: each (dataset, tokens) for the valid target tokens its steps commit, the last for the rest of the run (None), or for its tokens, where the data ends (the run’s last step). A stage ends at the step whose commit reaches its tokens, on every process alike, so its last step may commit past them; one that a single step’s commit passes reads nothing. A corpus that an earlier stage read continues from where it left it, and what the run had read ahead of the end is not trained on. Each stage is a whole chain, laid out and batched as it reads.

dataclass source

class ObjectiveData(
train: tuple[RecordStep | pygrain.RandomMapTransform | Expansion | Configured, ...] = (),
evaluate: tuple[RecordStep | Expansion | Configured, ...] = (),
pack: Pack | None = None,
packed: tuple[RecordStep, ...] = (),
train_batch: tuple[HostStep, ...] = (),
evaluate_batch: tuple[HostStep, ...] = (),
geometry: RecordStep | None = None,
)
geometry: RecordStep | None

The rows the objective reads where a chain lays out none of its own, after its other record steps; rows a chain packs or cuts into windows (Pack, Windows) keep the length it gives them, as a schedule’s stages each give theirs (dew.data.compile.objective_steps).

function source

def record_stage(
step: RecordStep | pygrain.RandomMapTransform | Expansion,
*,
identity: str | None = None,
configured: Bound | None = None,
) -> Map | RandomMap | FlatMap

The stage a record step is applied by where no verb names it, from what the step says it is: an expansion’s records (FlatMap), a generator’s draws (RandomMap), or one record’s replacement (Map), dropping records where it says it does (rejects). An objective’s steps and a configured step’s binding are read through it; a verb (map, random_map, filter_map, random_filter_map, flat_map) builds the stage its name states, which a step of another kind is refused.