dew.data.sources.pytorch
Map-style torch datasets, imported on use and read through the shared pipeline.
| Name | Summary |
|---|---|
TorchRecords | Reads a map-style torch dataset by index, each sample as named fields. |
Torch | A map-style torch dataset, with explicit content identity for plans and artifacts. |
from_torch | A map-style torch.utils.data.Dataset as a Dew dataset, its samples named by fields where they are tuples. |
TorchRecords
Section titled “TorchRecords”class TorchRecords(dataset: TorchDataset, names: Sequence[str] | None)Reads a map-style torch dataset by index, each sample as named fields.
A dict sample keeps its keys, and names names the values of a tuple
sample, or the one value of any other. Tensors come to the host as
numpy, and every value is a provider row’s field (fields). The repr
names the dataset as a saved position needs (describe).
class Torch( dataset: TorchDataset, fields: tuple[str, ...] | None = None, identity: str | None = None, known: Mapping[str, object] | None = None,)A map-style torch dataset, with explicit content identity for plans and artifacts.
known: Mapping[str, object] | None-
Facts of its records the caller states, which its metadata and
Dataset.factscarry.
Torch.stated
Section titled “Torch.stated”def stated() -> Mapping[str, object]Torch.label
Section titled “Torch.label”def label() -> strTorch.describe
Section titled “Torch.describe”def describe() -> strTorch.fingerprint
Section titled “Torch.fingerprint”def fingerprint() -> Mapping[str, object]Torch.supports
Section titled “Torch.supports”def supports(read: Read) -> boolTorch.records
Section titled “Torch.records”def records(read: Read) -> RecordsTorch.manifest
Section titled “Torch.manifest”def manifest(version: int) -> ShardManifestThe map-style records as a stream of consecutive index ranges (_indexed_manifest), as a
memory source streams.
Torch.metadata
Section titled “Torch.metadata”def metadata() -> Mapping[str, object]Torch.lengths
Section titled “Torch.lengths”def lengths() -> np.ndarray | Nonefrom_torch
Section titled “from_torch”def from_torch( dataset: TorchDataset, fields: tuple[str, ...] | None = None, *, identity: str | None = None,) -> TorchA map-style torch.utils.data.Dataset as a Dew dataset, its samples
named by fields where they are tuples. Plans and artifacts require a
versioned content identity, including non-exact resumes (Torch).