Skip to content

dew.data.plan

Execution identities and the order a saved record count names.

NameSummary
REBUILD
RECORD
SCHEDULEThe routes a run whose chain changed since it saved its position takes (PlanOrderChanged.routes): built as it was; resumed from a record that holds its chain (RunConfig.train names it where the saved run’s record does); a mix saved before its order was seeded read in that schedule (rows 3870, 3881), which a layer above spells as its own API reads it (PlanOrderChanged.routed, a decision mixture’s).
PlanOrderChangedA saved data position’s plan whose order is not the chain’s now: each path of the order changed (selection_seed, phases[1].selection_seed), its saved value and its value now (None where one holds none), and the routes its run goes on by (REBUILD, RECORD, SCHEDULE), a mix saved unseeded and seeded now, nothing else changed, naming SCHEDULE.
DigestA running hash, as hashlib’s are.
array_digestThe digest an array of dtype and shape is fingerprinted by, its header taken (dew._hashing.array_header, which tells apart dtypes of one byte layout): its elements’ bytes follow in C order, whole (fingerprint) or page by page as a published plan writes them (generation._write_plan), to the same hash.
fingerprintHash order metadata, including an array’s shape and dtype.
identifiedWhether a source snapshot states the content of every store in it: a store that holds no version of its own (a Grain dataset, an iterator, a torch dataset) states it by the name it was given, and one given none can’t say a later run reads the same records.
callable_namestep’s qualified name, module.qualname: a function’s or a bound method’s own, else its type’s, as a batch encoder names the reader it runs.
StepIdentityWhat a step computes (semantic), and the bytes of each external module its code comes from (modules): its own, then those of the factory that configured it.
ManifestThe dependency manifest shared by an execution plan and its cache.
same_orderWhether two orders, as a saved position and a stream state them in JSON, are one order: compared without their sources’ descriptions, which name where a host mounted them, as a pool compares its processes’ orders (Plan.identity).
unplacedorder without its sources’ descriptions, which name where a host mounted them, its corpora’s and phases’ included.
earlier_positionA per-share read’s saved position (its data plan and its Grain iterator’s state) as plan, this run’s, reads it: the plan as earlier_plan translates it, and the iterator’s state, where the plan was written before version 8, with the counts version 8 keeps that a Dew of version 7 may not have: of what a read delivered (each corpus’s delivered_rows, which one that counted each row as one example held as accepted), of its cuts by kind (CUT_COUNTS, none), of the examples its filter held pending (consumed, none), and of the examples refused past a held row and those a causal window credited (its carry of _dew_refused, none, beside its _dew_consumed, and credited, none).
COUNTERSThe format of a per-share position’s counters (dew.data.readers): 2 counts a mixture a run bounds in examples by member where its rows are delivered (row 3312), which a position of an earlier format, holding no counters, does not (_by_member); 3 counts the valid target tokens each corpus and member delivered (row 4119), which a position of format 2 does not (_tokens_counted).
read_otherwise_at_8Why version 8 reads a version 7 data plan’s streamed order otherwise, None where it reads it alike: where its read is not streamed, one reader reads it, no take reads its order (no Limit among its stages, its mixed corpora’s orders within it), or its takes are no bound in examples and no loading workers read them, a corpus’s own pool among them (pools).
read_otherwise_at_9Why version 9 reads a version 8 data plan’s order otherwise, None where it reads it alike: where a mixture’s corpus its order holds shuffles its records, at any depth, which version 9 draws by the corpus’s key in the mixture too, or where its chain read alone shuffles and plan, the chain as this run reads it, keys that shuffle by the chain’s name (named in its order), or a phase’s chain shuffles, which its own name may key; an unnamed chain’s shuffle alone draws by its seed alone, as 8 drew it (dew.data.lowering._Build.ordered, rows 3840, 4063).
read_otherwise_at_10Why version 10 reads a version 9 data plan’s order otherwise, None where it reads it alike: where its read streams a Parquet store (streams_parquet), whose shards 9 made of its row groups and 10 makes of its files (FILE_SHARDS), so its shards’ order, and the places a position names, are others.
earlier_plansaved, a data plan a saved position records, as plan, this run’s, reads it (Plan.check): the position’s one translation, where a reader’s set_state decodes it.
grown_limitsThe limits the order current sets higher than saved does, each its saved and its current record, where current is saved but for them; None where it differs otherwise.
FILE_SHARDSThe plan version from which a streamed Parquet store’s shards are its files, each file’s row groups read as its reader reaches them and its footer read as it opens, cut into its row groups only where readers and workers outnumber its files (dew.data.sources.files.Files.manifest): 9’s shards were its row groups, numbered across every file, so a stream read every file’s footer before its first record (FineWeb-Edu’s default configuration’s 2410 files, 196 s, row 4131).
PLAN_VERSIONSThe versions this Dew reads a plan into rows by, the latest a new run’s: 9 folds a corpus’s key, its name in the mixture that reads it or a named chain’s own, into each shuffle of its own, so corpora of one length shuffled by one seed read in orders of their own and a named corpus reads one order mixed or alone (row 3840), where 8 read them in one permutation; 10 streams a Parquet store by its files (FILE_SHARDS).
read_byversion, refused where this Dew reads by none of it (PLAN_VERSIONS): what every read (dew.data.compile’s compile, iterate and publish) asks before it prepares anything, where another would read as the version it falls among (11 as 10, 5 as 8).
filled_versionThe plan version whose shards a streamed fill numbers its rows by, from its cache key’s fill (Manifest.fill): FILE_SHARDS where it names a Parquet store’s shards its files, the version before it otherwise, as every other store’s shards are alike at both (dew.data.sources.files.Files.manifest).
streams_parquetWhether source, a store’s fingerprint as a plan records it (Manifest.semantic’s source), names a Parquet store at any depth (a selection’s, a concatenation’s part, a mixture’s corpus): the stores whose streamed shards a plan version decides (FILE_SHARDS).
PlanOrder-defining fields checked before a restored iterator opens readers.

attribute source

REBUILD = 'build its chain as it was'

attribute source

RECORD = 'resume it from its record, which holds the chain it read (`dew train <run directory>/run.json`)'

attribute source

SCHEDULE = 'read its mix unseeded, in the schedule it saved (`mix(..., seed=None)`)'

The routes a run whose chain changed since it saved its position takes (PlanOrderChanged.routes): built as it was; resumed from a record that holds its chain (RunConfig.train names it where the saved run’s record does); a mix saved before its order was seeded read in that schedule (rows 3870, 3881), which a layer above spells as its own API reads it (PlanOrderChanged.routed, a decision mixture’s).

class source

class PlanOrderChanged(
changed: Mapping[str, tuple[JSON | None, JSON | None]],
routes: tuple[str, ...],
)

A saved data position’s plan whose order is not the chain’s now: each path of the order changed (selection_seed, phases[1].selection_seed), its saved value and its value now (None where one holds none), and the routes its run goes on by (REBUILD, RECORD, SCHEDULE), a mix saved unseeded and seeded now, nothing else changed, naming SCHEDULE.

def between(saved: Mapping[str, object], now: Mapping[str, object]) -> PlanOrderChanged

The refusal of a position saved under order saved, read under now, as the check compares them (unplaced, Plan.check): each path whose value differs, a mix’s corpora, a phase’s and a mix of mixes’ members by their places (row 3914).

def routed(*, recorded: bool, unseeded: str = SCHEDULE) -> PlanOrderChanged

This refusal with the record route first where the saved run’s record holds its chain, and the schedule’s route as the run’s own layer spells it (unseeded).

class source

class Digest(Protocol)

A running hash, as hashlib’s are.

def update(chunk: bytes, /) -> None
def hexdigest() -> str

function source

def array_digest(dtype: np.dtype, shape: Sequence[int]) -> Digest

The digest an array of dtype and shape is fingerprinted by, its header taken (dew._hashing.array_header, which tells apart dtypes of one byte layout): its elements’ bytes follow in C order, whole (fingerprint) or page by page as a published plan writes them (generation._write_plan), to the same hash.

function source

def fingerprint(value: JSON | Mapping[str, object] | np.ndarray) -> str

Hash order metadata, including an array’s shape and dtype.

function source

def identified(
value: dict[str, JSON] | list[JSON] | str | int | float | bool | None,
) -> bool

Whether a source snapshot states the content of every store in it: a store that holds no version of its own (a Grain dataset, an iterator, a torch dataset) states it by the name it was given, and one given none can’t say a later run reads the same records.

function source

def callable_name(step: Callable[..., Batch]) -> str

step’s qualified name, module.qualname: a function’s or a bound method’s own, else its type’s, as a batch encoder names the reader it runs. A step’s identity (StepIdentity.of) names a bound method by its type, as plans have always written it; only a function is named by its own name there.

dataclass source

class StepIdentity(
name: str,
config: str,
exact: bool,
modules: tuple[str, ...] = (),
declared: str | None = None,
dependencies: Mapping[str, object] = dict(),
code: tuple[int, ...] = (),
)

What a step computes (semantic), and the bytes of each external module its code comes from (modules): its own, then those of the factory that configured it. code places among modules those of a function outside Dew that declares no identity, which state nothing of what it computes but its code (Plan.check); no record holds it.

def of(
step: object,
identity: str | None = None,
*,
cache: bool = False,
) -> StepIdentity

step’s identity, refused where a cache keys it (cache) and cannot (keyable).

def record() -> dict[str, JSON]

The identity as a manifest records it: every field but code, which its modules’ places say.

def composed(factory: StepIdentity, config: str) -> StepIdentity

This step’s identity as the Configured stage factory resolved it, under config: the factory’s modules after its own, and their places of code (code) with them.

def keyable(*, function: bool, external: bool) -> bool

Whether a cache can key what the step computes: a stable identity, declared wherever the step is a function or its code is outside Dew, which Dew’s revision does not name.

def semantic() -> dict[str, JSON]

What the step computes, on any host: its identity without its modules’ bytes, which only a manifest’s environment records.

dataclass source

class Manifest(
source: Mapping[str, object],
steps: tuple[StepIdentity, ...] = (),
ran: tuple[StepIdentity, ...] = (),
revision: str = 'unversioned',
lock: str = '',
python: str = platform.python_version(),
platform: str = platform.platform(),
tokenizer: Mapping[str, object] = dict(),
fill: Mapping[str, object] = dict(),
codec: int = 1,
schema: int = 1,
clean: bool = True,
)

The dependency manifest shared by an execution plan and its cache.

ran are the steps a memory read already ran into a source it holds. That source’s snapshot names them by what they compute, so the semantic record leaves them out and the environment takes their modules. They are steps of the plan all the same, so the manifest is exact only when they are as well as its own.

exact: bool

Whether a later run can tell it makes these rows again: every step has a stable identity and every store states its content (identified).

def of(
source: Mapping[str, object],
steps: Sequence[StepIdentity],
*,
ran: Sequence[StepIdentity] = (),
fill: Mapping[str, object] | None = None,
) -> Manifest
def record() -> dict[str, JSON]
def semantic() -> dict[str, JSON]
def environment() -> dict[str, JSON]

function source

def same_order(saved: str, current: str) -> bool

Whether two orders, as a saved position and a stream state them in JSON, are one order: compared without their sources’ descriptions, which name where a host mounted them, as a pool compares its processes’ orders (Plan.identity).

function source

def unplaced(
order: dict[str, JSON] | list[JSON] | str | int | float | bool | None,
) -> JSON

order without its sources’ descriptions, which name where a host mounted them, its corpora’s and phases’ included.

function source

def earlier_position(
saved: Mapping[str, object],
plan: Plan,
) -> tuple[Mapping[str, object], JSON]

A per-share read’s saved position (its data plan and its Grain iterator’s state) as plan, this run’s, reads it: the plan as earlier_plan translates it, and the iterator’s state, where the plan was written before version 8, with the counts version 8 keeps that a Dew of version 7 may not have: of what a read delivered (each corpus’s delivered_rows, which one that counted each row as one example held as accepted), of its cuts by kind (CUT_COUNTS, none), of the examples its filter held pending (consumed, none), and of the examples refused past a held row and those a causal window credited (its carry of _dew_refused, none, beside its _dew_consumed, and credited, none). The position’s one translation; carried to 1.0 (ledger 1552).

attribute source

COUNTERS = 3

The format of a per-share position’s counters (dew.data.readers): 2 counts a mixture a run bounds in examples by member where its rows are delivered (row 3312), which a position of an earlier format, holding no counters, does not (_by_member); 3 counts the valid target tokens each corpus and member delivered (row 4119), which a position of format 2 does not (_tokens_counted).

function source

def read_otherwise_at_8(saved: Mapping[str, object]) -> str | None

Why version 8 reads a version 7 data plan’s streamed order otherwise, None where it reads it alike: where its read is not streamed, one reader reads it, no take reads its order (no Limit among its stages, its mixed corpora’s orders within it), or its takes are no bound in examples and no loading workers read them, a corpus’s own pool among them (pools). Version 7 read a take before its loading workers (a .take) on each of them, and a run’s bound in examples (its manifest’s MarkConsumed step) whole on each of a pool’s shares and their workers; version 8 takes the one across them and deals the other to them. So a position 7 wrote of such a read can go on only as 7 read it, which reads the bound once a reader: the over-read 8 ended.

function source

def read_otherwise_at_9(saved: Mapping[str, object], plan: Plan) -> str | None

Why version 9 reads a version 8 data plan’s order otherwise, None where it reads it alike: where a mixture’s corpus its order holds shuffles its records, at any depth, which version 9 draws by the corpus’s key in the mixture too, or where its chain read alone shuffles and plan, the chain as this run reads it, keys that shuffle by the chain’s name (named in its order), or a phase’s chain shuffles, which its own name may key; an unnamed chain’s shuffle alone draws by its seed alone, as 8 drew it (dew.data.lowering._Build.ordered, rows 3840, 4063).

function source

def read_otherwise_at_10(saved: Mapping[str, object]) -> str | None

Why version 10 reads a version 9 data plan’s order otherwise, None where it reads it alike: where its read streams a Parquet store (streams_parquet), whose shards 9 made of its row groups and 10 makes of its files (FILE_SHARDS), so its shards’ order, and the places a position names, are others.

function source

def earlier_plan(saved: Mapping[str, object], plan: Plan) -> Mapping[str, object]

saved, a data plan a saved position records, as plan, this run’s, reads it (Plan.check): the position’s one translation, where a reader’s set_state decodes it. A plan of version 7 reads as 8, one of 8 as 9 and one of 9 as 10, where the later reads its order alike (read_otherwise_at_8, read_otherwise_at_9, read_otherwise_at_10). A token store a Dew before row 3028 named without saying how its documents are read is named as today’s names it where that Dew read the same documents of it (_named_as_today); its derived order holds today’s lengths (plan’s) of each source that Dew stated none for (_unstated_lengths), where the plan names the source as today’s does; the saved and current manifests’ sources are numbered as their derived facts number them. Carried to 1.0 (ledger 1552).

function source

def grown_limits(
saved: dict[str, JSON] | list[JSON] | str | int | float | bool | None,
current: dict[str, JSON] | list[JSON] | str | int | float | bool | None,
) -> frozenset[tuple[str, str]] | None

The limits the order current sets higher than saved does, each its saved and its current record, where current is saved but for them; None where it differs otherwise. Only the run’s own chain’s limit grows (never a mixed corpus’s or a phase’s, which the mixture’s own stages read after it), and only where nothing after it but a batch reads the records it holds, as the examples clock’s bound (dew.data.compile._bounded) of a chain that neither packs nor windows: its first records are the saved one’s, read further, where a repeat, a shuffle, a split, a pack or windows after it would read them anew (row 2649).

attribute source

FILE_SHARDS = 10

The plan version from which a streamed Parquet store’s shards are its files, each file’s row groups read as its reader reaches them and its footer read as it opens, cut into its row groups only where readers and workers outnumber its files (dew.data.sources.files.Files.manifest): 9’s shards were its row groups, numbered across every file, so a stream read every file’s footer before its first record (FineWeb-Edu’s default configuration’s 2410 files, 196 s, row 4131).

attribute source

PLAN_VERSIONS = (8, 9, FILE_SHARDS)

The versions this Dew reads a plan into rows by, the latest a new run’s: 9 folds a corpus’s key, its name in the mixture that reads it or a named chain’s own, into each shuffle of its own, so corpora of one length shuffled by one seed read in orders of their own and a named corpus reads one order mixed or alone (row 3840), where 8 read them in one permutation; 10 streams a Parquet store by its files (FILE_SHARDS). A run saved under an earlier version reads by it (carried to 1.0, ledger 1552).

function source

def read_by(version: int) -> int

version, refused where this Dew reads by none of it (PLAN_VERSIONS): what every read (dew.data.compile’s compile, iterate and publish) asks before it prepares anything, where another would read as the version it falls among (11 as 10, 5 as 8).

function source

def filled_version(fill: Mapping[str, object]) -> int

The plan version whose shards a streamed fill numbers its rows by, from its cache key’s fill (Manifest.fill): FILE_SHARDS where it names a Parquet store’s shards its files, the version before it otherwise, as every other store’s shards are alike at both (dew.data.sources.files.Files.manifest).

function source

def streams_parquet(source: JSON) -> bool

Whether source, a store’s fingerprint as a plan records it (Manifest.semantic’s source), names a Parquet store at any depth (a selection’s, a concatenation’s part, a mixture’s corpus): the stores whose streamed shards a plan version decides (FILE_SHARDS).

dataclass source

class Plan(
lowering: Literal['global', 'per-share'],
manifest: Manifest,
order: Mapping[str, object],
derived: Mapping[str, str] = dict(),
topology: Mapping[str, object] = dict(),
rejections: Literal['drop', 'mask'] = 'drop',
cache: tuple[str, ...] = (),
volatile: bool = False,
members: Mapping[str, str] = dict(),
stores: tuple[TokenFiles, ...] = (),
earlier_members: Mapping[str, tuple[JSON, tuple[TokenFiles, ...]]] = dict(),
version: int = PLAN_VERSIONS[-1],
)

Order-defining fields checked before a restored iterator opens readers.

version names how Dew reads a plan into rows, which its fields do not say: a change that makes other rows or draws from the same plan raises it, and a position saved under another version is refused.

members: Mapping[str, str]

A mixture’s members, each what it reads and runs as one digest under its key, and its own stages under the empty key (dew.data.analysis._member_manifests): what a streamed mix’s resume reads of a member that joined or left (check).

stores: tuple[TokenFiles, ...]

The token stores beneath the plan’s sources, which a plan a Dew before row 3028 saved names otherwise (earlier_plan); no part of the plan’s record.

earlier_members: Mapping[str, tuple[JSON, tuple[TokenFiles, ...]]]

Each of members’ manifests, undigested, with the token stores the member reads, from which a resume makes the digests an earlier Dew made of it (earlier_plan); no part of the plan’s record.

version: int

The version the plan is read into rows by (PLAN_VERSIONS): this Dew’s, or the one a run saved under, which reads it as it did (dew.data.compile.compile’s plan_version).

def record() -> dict[str, JSON]
def identity() -> str

What every process of a pool reads, which the pool agrees before any reads: the order, as a reader can follow it, then the rest of the plan as its parts’ digests (parts), so a pool that disagrees is shown which part differs. The order’s sources are named by the manifest’s content, not by their descriptions: hosts that mount one dataset at different paths read the same records.

def parts() -> dict[str, JSON]

A digest of each part of the plan but its order, its environment and the share this process reads, which only the position it saves keeps: the manifest’s parts each apart, and a list’s entries, such as each source’s content and each step’s identity, one by one.

def check(
saved: Mapping[str, object],
*,
resume: Literal['exact', 'non-exact'] = 'exact',
) -> tuple[str, ...]

Refuse a saved plan whose rows this plan does not make again: another version, order, derived order fact, per-share topology or semantic manifest (Manifest.semantic), or, resumed exactly, a step outside Dew whose code changed and that declares no identity. A changed environment (Manifest.environment) is returned and warned of, and refused for a per-share plan.