dew.data.sources
Storage capabilities and the source boundary every chain reads through.
| Name | Summary |
|---|---|
TFDS | A prepared TFDS split, read with TFDS’s native ArrayRecord data source. Documented in dew.data.sources.tfds. |
Concatenate | The records of sources one store after another, as one store: two splits of one Hub dataset read as one (Concatenate((hub.with_split("train"), hub.with_split("validation")))), or stores of any kinds. |
Files | Files at URLs, with producer indexes or explicit provisioning for text. Documented in dew.data.sources.files. |
Hub | A pinned split of a Hub dataset: name is its configuration, as datasets.load_dataset(repo, name=) takes it, and columns the fields each record keeps (all where None). Documented in dew.data.sources.hub. |
Memory | Records, column arrays, or a DataFrame or Arrow table (DataFrame), with immutable positional IDs. Documented in dew.data.sources.memory. |
Read | How a run reads every source it names: memory reads each whole into RAM (and may materialize its deterministic stages, design §10.3), indexed reads records by index through an index the store holds, streamed reads shards in order with no index. Documented in dew.data.sources.base. |
Shard | One piece of a streamed source: a file, a byte or row range of one, a Parquet row group, or a disjoint row share. Documented in dew.data.sources.base. |
ShardManifest | A streamed source’s shards, in canonical order, and the reader that opens one: a picklable function of a shard and the grain ReadOptions its reading reads it under (None for Grain’s own), returning a checkpointable grain IterDataset over the shard’s records in order. Documented in dew.data.sources.base. |
Source | A store a pipeline reads, and the read modes it supports. Documented in dew.data.sources.base. |
TokenFiles | A committed token export (dew.data.text.export_tokens, dew data export --tokenizer): its documents of token ids, the metadata its manifest states and where its documents end, each a record its producer wrote, as its length index states it (document_ends). |
from_grain | dataset as a Dew dataset, its content version identity where given (Grain). Documented in dew.data.sources.grain. |
from_hf | A Hugging Face datasets.Dataset or IterableDataset as a Dew dataset (HF): an IterableDataset with its identity, unshuffled. Documented in dew.data.sources.hf. |
from_iter | The records factory() iterates as a Dew dataset, its content version identity where given, resumed through the iterator’s own state and load_state where it has them, or by replay where it declares replay (Iterated). Documented in dew.data.sources.iterator. |
from_torch | A map-style torch.utils.data.Dataset as a Dew dataset, its samples named by fields where they are tuples. Documented in dew.data.sources.pytorch. |
source | The store value names: a Source as it is; a tfds://name[:version] or hf://owner/repo[@revision] URL as its TFDS or Hub dataset, split its split (“train” by default); any other URL as its files (Files, format naming their record format where the URL’s suffix does not), or as a committed token export (TokenFiles) where its manifest says it is one; and records in memory (a sequence of rows, a mapping of columns or a data frame) as Memory. |
Concatenate
Section titled “Concatenate”class Concatenate(sources: tuple[Source, ...])The records of sources one store after another, as one store: two splits of one Hub
dataset read as one (Concatenate((hub.with_split("train"), hub.with_split("validation")))),
or stores of any kinds. A record’s position is its place in that order, every source’s shards
are its shards, and its identity is its sources’ in order, so a run resumes on it as on any
store and a changed source refuses the resume as it would alone. It is a store wherever one is
read; concat joins stores as a dataset.
Concatenate.label
Section titled “Concatenate.label”def label() -> strConcatenate.describe
Section titled “Concatenate.describe”def describe() -> strConcatenate.consumed
Section titled “Concatenate.consumed”def consumed(read: Read) -> ConcatenateConcatenate.fingerprint
Section titled “Concatenate.fingerprint”def fingerprint() -> Mapping[str, object]Concatenate.supports
Section titled “Concatenate.supports”def supports(read: Read) -> boolConcatenate.records
Section titled “Concatenate.records”def records(read: Read) -> RecordsConcatenate.metadata
Section titled “Concatenate.metadata”def metadata() -> Mapping[str, object]What every source states of its records, as one store’s: the counts (COUNTED) added
where each source states one, the tokenizer contract where the sources that state it agree
(tokenizer_contract, which refuses two), the rules any part’s rows were counted by
where every part states its own (count_rules), and any other fact, class names among
them, where every source states the one value.
Concatenate.stated
Section titled “Concatenate.stated”def stated() -> Mapping[str, object]What every part states of its records without reading them, merged as metadata is.
Concatenate.lengths
Section titled “Concatenate.lengths”def lengths() -> np.ndarray | NoneConcatenate.manifest
Section titled “Concatenate.manifest”def manifest(version: int) -> ShardManifestEach store’s shards at version, one store after another: their index order where each
store’s is, as records joins them; their rows counted (ShardManifest.sized) where every
store’s are stated or counted.
TokenFiles
Section titled “TokenFiles”class TokenFiles( url: str, storage_options: Mapping[str, object] | None = None, field: str | None = None,)A committed token export (dew.data.text.export_tokens, dew data export --tokenizer): its
documents of token ids, the metadata its manifest states and where its documents end, each a record
its producer wrote, as its length index states it (document_ends). An export holds one split;
validation tokens are an export of their own.
committed: Mapping[str, object]-
The export’s committed manifest, read through this store’s own storage options.
TokenFiles.label
Section titled “TokenFiles.label”def label() -> strTokenFiles.describe
Section titled “TokenFiles.describe”def describe() -> strTokenFiles.stated
Section titled “TokenFiles.stated”def stated() -> Mapping[str, object]What the store’s manifest says of its tokens, none of them read (Dataset.facts): their
tokenizer, vocabulary and EOS, as token ids under “text”; an export not here states nothing.
TokenFiles.metadata
Section titled “TokenFiles.metadata”def metadata() -> Mapping[str, object]TokenFiles.fingerprint
Section titled “TokenFiles.fingerprint”def fingerprint() -> Mapping[str, object]TokenFiles.read_alike
Section titled “TokenFiles.read_alike”def read_alike() -> boolWhether a Dew before row 3028, which read a token store’s documents by its EOS, read this store’s
as its producer’s records: stating no EOS, one record holding every token; stating one, each record
ended by that EOS and holding no other (_earlier_ends), a scan of every token of the field read,
kept in Dew’s cache under the export’s content (export_content), which no write changes, and that
field. Read only to resume a plan that Dew saved (dew.data.plan.earlier_plan). Carried to 1.0
(ledger 1552).
TokenFiles.supports
Section titled “TokenFiles.supports”def supports(read: Read) -> boolTokenFiles.can_provision
Section titled “TokenFiles.can_provision”def can_provision(read: Read) -> boolTokenFiles.preparation_bytes
Section titled “TokenFiles.preparation_bytes”def preparation_bytes() -> intNone to read: the producer’s length index states every document’s end.
TokenFiles.provision
Section titled “TokenFiles.provision”def provision() -> TokenFilesNothing to write: the producer’s length index, which every export carries, states where each
document ends (document_ends).
TokenFiles.records
Section titled “TokenFiles.records”def records(read: Read) -> RecordsEach document by index (document_ends), in every read.
TokenFiles.document_ends
Section titled “TokenFiles.document_ends”def document_ends() -> np.ndarrayWhere each document ends among the export’s tokens: each record its producer wrote, by its
length index (TokenRecords.shards), a shard’s after the shard before it, an EOS its tokens spell
within it or not, as the documents it exported were (dew.data.text.export_tokens). An empty
record holds no document, so none is read of it.
TokenFiles.manifest
Section titled “TokenFiles.manifest”def manifest(version: int) -> ShardManifestTokenFiles.lengths
Section titled “TokenFiles.lengths”def lengths() -> np.ndarraysource
Section titled “source”def source( value: object, *, format: Format | None = None, split: str | None = None, provision: bool = False,) -> SourceThe store value names: a Source as it is; a tfds://name[:version] or hf://owner/repo[@revision]
URL as its TFDS or Hub dataset, split its split (“train” by default); any other URL as its files
(Files, format naming their record format where the URL’s suffix does not), or as a committed
token export (TokenFiles) where its manifest says it is one; and records in memory (a sequence
of rows, a mapping of columns or a data frame) as Memory. Files and records read what they
hold, whatever split is named: a split is a Hub or TFDS URL’s. With provision, the store builds
the indexes its reads need first.