Skip to content

dew.data.sources.base

What a pipeline reads from: a store, its identity, and how it can be read.

A Source names a store (records in memory, files at a URL, a Hub dataset, a prepared TFDS build, a torch dataset, a catalogue of URLs) and says which of the three read modes it supports (Read). Memory and indexed reads hand over records by index (dew.data.dataset.Records); a streamed read hands over a ShardManifest, the store’s physical shards with one reader each, which the pipeline slices over its readers and workers and interleaves.

A source states facts about itself (metadata, lengths) and never scans to find them: a step that needs a global fact reads it here or from config, and a streamed source without it is refused before the run opens. describe and fingerprint are its identity: the first is what a saved position names, the second what a cache’s key and a run’s plan record (the URI, the resolved revision, each file’s size and ETag). label is the short name a person reads in a run’s metrics, never its identity.

NameSummary
READS
ConsumedA source whose read of a given kind goes through another source.
ReadHow a run reads every source it names: memory reads each whole into RAM (and may materialize its deterministic stages, design §10.3), indexed reads records by index through an index the store holds, streamed reads shards in order with no index.
ShardOne piece of a streamed source: a file, a byte or row range of one, a Parquet row group, or a disjoint row share.
ShardManifestA streamed source’s shards, in canonical order, and the reader that opens one: a picklable function of a shard and the grain ReadOptions its reading reads it under (None for Grain’s own), returning a checkpointable grain IterDataset over the shard’s records in order.
SourceA store a pipeline reads, and the read modes it supports.

attribute source

READS: tuple[Read, ...] = ('memory', 'indexed', 'streamed')

class source

class Consumed(Protocol)

A source whose read of a given kind goes through another source.

Wrappers rebuild over their consumed children, retaining their own policy.

def consumed(read: Read) -> Source

The source a read of this one reads its records from.

attribute source

How a run reads every source it names: memory reads each whole into RAM (and may materialize its deterministic stages, design §10.3), indexed reads records by index through an index the store holds, streamed reads shards in order with no index.

dataclass source

class Shard(
id: str,
uri: str,
start: int = 0,
stop: int | None = None,
rows: int | None = None,
)

One piece of a streamed source: a file, a byte or row range of one, a Parquet row group, or a disjoint row share. id is immutable across runs (path and range, or a Hub revision and path), so a cache keys rows by (id, row).

dataclass source

class ShardManifest(
shards: tuple[Shard, ...],
reader: ShardOpener,
subdivide: Callable[[int], tuple[Shard, ...]] | None = None,
index_ordered: bool = False,
counted: Callable[[int], tuple[int, ...]] | None = None,
fixed_parts: bool = False,
)

A streamed source’s shards, in canonical order, and the reader that opens one: a picklable function of a shard and the grain ReadOptions its reading reads it under (None for Grain’s own), returning a checkpointable grain IterDataset over the shard’s records in order.

When physical shards cannot fill the readers and workers, subdivide receives the needed count and returns exactly that many Shards, each a disjoint part of these records with an exact per-part position read by the same reader. None keeps the physical-shard requirement.

index_ordered states that the shards, one after another, read the records in their index order, as a store read by index streams them (_indexed_manifest).

counted, where the shards state no rows that their store can count, returns each one’s rows, given the reads it may make at once, which a read that needs them reads then (sized): a streamed Parquet store’s files’ from their footers, which a read that needs none never reads.

fixed_parts states that subdivide cuts the same parts whatever its count, the store’s own finer shards (a Parquet store’s row groups): a stream read by them keys its fill by them, which its publication reads as the store’s (parts, dew.data.generation).

def parts() -> tuple[Shard, ...]

The store’s finer shards its subdivision always cuts (fixed_parts), as a read by more readers than its shards cuts them; none where it cuts by count, or cuts none.

def parted(reads: int) -> dict[str, tuple[Shard, ...]]

Each shard’s parts (parts), in order, by the shard’s id: the run of parts whose rows make its own (sized, reads at once), every shard’s in turn; none where a shard or a part states no rows (a filtered store’s), or the parts make no shard’s rows.

def sized(reads: int) -> ShardManifest

The manifest with each shard’s rows its store counts (counted), reads at once (its reading’s Loading.threads), as a read that needs them reads them (a bound in examples dealt by its shards’ rows, a read placed by index, a pass’s count); itself where its store counts none.

class source

class Source(Protocol)

A store a pipeline reads, and the read modes it supports.

def describe() -> str

A stable description: store, URI, revision and split, never an address.

def label() -> str

The store’s short name in a run’s metrics: its dataset, builder or file name.

def fingerprint() -> Mapping[str, object]

The store’s identity for a cache key and a run’s plan: the URI, the resolved revision, and each file’s size and ETag or generation.

def supports(read: Read) -> bool

Whether the source can be read this way without provisioning.

def records(read: Read) -> Records

The records by index, for a memory or an indexed read.

def manifest(version: int) -> ShardManifest

The shards and their reader, for a streamed read of data plan version (dew.data.plan.PLAN_VERSIONS), which decides a store’s shards where versions shard it otherwise (a Parquet store’s, dew.data.plan.FILE_SHARDS).

def metadata() -> Mapping[str, object]

Facts the store states about itself (class names, a mixture’s shares, slot maxima, per-corpus token counts), never found by a scan.

def lengths() -> np.ndarray | None

Each record’s length when the store states it (a producer’s index), which length grouping ranks by; None when it does not.