dew.data.sources.base
What a pipeline reads from: a store, its identity, and how it can be read.
A Source names a store (records in memory, files at a URL, a Hub dataset,
a prepared TFDS build, a torch dataset, a catalogue of URLs) and says which of
the three read modes it supports (Read). Memory and indexed reads hand over
records by index (dew.data.dataset.Records); a streamed read hands over a
ShardManifest, the store’s physical shards with one reader each, which the
pipeline slices over its readers and workers and interleaves.
A source states facts about itself (metadata, lengths) and never scans to
find them: a step that needs a global fact reads it here or from config, and
a streamed source without it is refused before the run opens. describe and
fingerprint are its identity: the first is what a saved position names, the
second what a cache’s key and a run’s plan record (the URI, the resolved
revision, each file’s size and ETag). label is the short name a person reads
in a run’s metrics, never its identity.
| Name | Summary |
|---|---|
READS | |
Consumed | A source whose read of a given kind goes through another source. |
Read | How a run reads every source it names: memory reads each whole into RAM (and may materialize its deterministic stages, design §10.3), indexed reads records by index through an index the store holds, streamed reads shards in order with no index. |
Shard | One piece of a streamed source: a file, a byte or row range of one, a Parquet row group, or a disjoint row share. |
ShardManifest | A streamed source’s shards, in canonical order, and the reader that opens one: a picklable function of a shard and the grain ReadOptions its reading reads it under (None for Grain’s own), returning a checkpointable grain IterDataset over the shard’s records in order. |
Source | A store a pipeline reads, and the read modes it supports. |
READS: tuple[Read, ...] = ('memory', 'indexed', 'streamed')Consumed
Section titled “Consumed”class Consumed(Protocol)A source whose read of a given kind goes through another source.
Wrappers rebuild over their consumed children, retaining their own policy.
Consumed.consumed
Section titled “Consumed.consumed”def consumed(read: Read) -> SourceThe source a read of this one reads its records from.
How a run reads every source it names: memory reads each whole into RAM
(and may materialize its deterministic stages, design §10.3), indexed reads
records by index through an index the store holds, streamed reads shards in
order with no index.
class Shard( id: str, uri: str, start: int = 0, stop: int | None = None, rows: int | None = None,)One piece of a streamed source: a file, a byte or row range of one,
a Parquet row group, or a disjoint row share. id is immutable across runs
(path and range, or a Hub revision and path), so a cache keys rows by (id, row).
ShardManifest
Section titled “ShardManifest”class ShardManifest( shards: tuple[Shard, ...], reader: ShardOpener, subdivide: Callable[[int], tuple[Shard, ...]] | None = None, index_ordered: bool = False, counted: Callable[[int], tuple[int, ...]] | None = None, fixed_parts: bool = False,)A streamed source’s shards, in canonical order, and the reader that
opens one: a picklable function of a shard and the grain ReadOptions its
reading reads it under (None for Grain’s own), returning a checkpointable
grain IterDataset over the shard’s records in order.
When physical shards cannot fill the readers and workers, subdivide
receives the needed count and returns exactly that many Shards, each
a disjoint part of these records with an exact per-part position read
by the same reader. None keeps the physical-shard requirement.
index_ordered states that the shards, one after another, read the
records in their index order, as a store read by index streams them
(_indexed_manifest).
counted, where the shards state no rows that their store can count,
returns each one’s rows, given the reads it may make at once, which a read
that needs them reads then (sized): a streamed Parquet store’s files’
from their footers, which a read that needs none never reads.
fixed_parts states that subdivide cuts the same parts whatever its
count, the store’s own finer shards (a Parquet store’s row groups): a
stream read by them keys its fill by them, which its publication reads as
the store’s (parts, dew.data.generation).
ShardManifest.parts
Section titled “ShardManifest.parts”def parts() -> tuple[Shard, ...]The store’s finer shards its subdivision always cuts (fixed_parts), as a read by more readers
than its shards cuts them; none where it cuts by count, or cuts none.
ShardManifest.parted
Section titled “ShardManifest.parted”def parted(reads: int) -> dict[str, tuple[Shard, ...]]Each shard’s parts (parts), in order, by the shard’s id: the run of parts whose rows make its
own (sized, reads at once), every shard’s in turn; none where a shard or a part states no rows
(a filtered store’s), or the parts make no shard’s rows.
ShardManifest.sized
Section titled “ShardManifest.sized”def sized(reads: int) -> ShardManifestThe manifest with each shard’s rows its store counts (counted), reads at once (its reading’s
Loading.threads), as a read that needs them reads them (a bound in examples dealt by its shards’
rows, a read placed by index, a pass’s count); itself where its store counts none.
Source
Section titled “Source”class Source(Protocol)A store a pipeline reads, and the read modes it supports.
Source.describe
Section titled “Source.describe”def describe() -> strA stable description: store, URI, revision and split, never an address.
Source.label
Section titled “Source.label”def label() -> strThe store’s short name in a run’s metrics: its dataset, builder or file name.
Source.fingerprint
Section titled “Source.fingerprint”def fingerprint() -> Mapping[str, object]The store’s identity for a cache key and a run’s plan: the URI, the resolved revision, and each file’s size and ETag or generation.
Source.supports
Section titled “Source.supports”def supports(read: Read) -> boolWhether the source can be read this way without provisioning.
Source.records
Section titled “Source.records”def records(read: Read) -> RecordsThe records by index, for a memory or an indexed read.
Source.manifest
Section titled “Source.manifest”def manifest(version: int) -> ShardManifestThe shards and their reader, for a streamed read of data plan version
(dew.data.plan.PLAN_VERSIONS), which decides a store’s shards where
versions shard it otherwise (a Parquet store’s, dew.data.plan.FILE_SHARDS).
Source.metadata
Section titled “Source.metadata”def metadata() -> Mapping[str, object]Facts the store states about itself (class names, a mixture’s shares, slot maxima, per-corpus token counts), never found by a scan.
Source.lengths
Section titled “Source.lengths”def lengths() -> np.ndarray | NoneEach record’s length when the store states it (a producer’s index), which length grouping ranks by; None when it does not.