dew.data.sources.iterator
Any resumable iterator read as a Dew store.
| Name | Summary |
|---|---|
Iterated | Records from factory(), a fresh iterator each call, read as one stream. |
from_iter | The records factory() iterates as a Dew dataset, its content version identity where given, resumed through the iterator’s own state and load_state where it has them, or by replay where it declares replay (Iterated). |
Iterated
Section titled “Iterated”class Iterated( factory: Callable[[], Iterator[Batch]], identity: str | None = None, state: Callable[[Iterator[Batch]], Value] | None = None, load_state: Callable[[Iterator[Batch], Value], None] | None = None, replay: bool = False, place: int = unnamed(),)Records from factory(), a fresh iterator each call, read as one
stream. Its position is what state(iterator) returns and
load_state(iterator, saved) restores, as WebDataset, torchdata,
Mosaic Streaming and Ray iterators each keep their own. replay
declares instead that factory() reads the same records each time it
opens, and a resume reads them again to its position (Replayed); a
stream with neither trains, and its resume is refused. identity declares
its content version in a run’s plan and its logs; with none, a resume
needs resume="non-exact" (Manifest.exact).
Iterated.label
Section titled “Iterated.label”def label() -> strIterated.describe
Section titled “Iterated.describe”def describe() -> strIterated.fingerprint
Section titled “Iterated.fingerprint”def fingerprint() -> Mapping[str, object]Iterated.supports
Section titled “Iterated.supports”def supports(read: Read) -> boolIterated.records
Section titled “Iterated.records”def records(read: Read) -> RecordsIterated.manifest
Section titled “Iterated.manifest”def manifest(version: int) -> ShardManifestIterated.metadata
Section titled “Iterated.metadata”def metadata() -> Mapping[str, object]Iterated.lengths
Section titled “Iterated.lengths”def lengths() -> np.ndarray | Nonefrom_iter
Section titled “from_iter”def from_iter( factory: Callable[[], Iterator[Batch]], *, identity: str | None = None, state: Callable[[Iterator[Batch]], Value] | None = None, load_state: Callable[[Iterator[Batch], Value], None] | None = None, replay: bool = False,) -> IteratedThe records factory() iterates as a Dew dataset, its content version identity where given, resumed
through the iterator’s own state and load_state where it has them,
or by replay where it declares replay (Iterated).