Skip to content

dew.data.sources.iterator

Any resumable iterator read as a Dew store.

NameSummary
IteratedRecords from factory(), a fresh iterator each call, read as one stream.
from_iterThe records factory() iterates as a Dew dataset, its content version identity where given, resumed through the iterator’s own state and load_state where it has them, or by replay where it declares replay (Iterated).

dataclass source

class Iterated(
factory: Callable[[], Iterator[Batch]],
identity: str | None = None,
state: Callable[[Iterator[Batch]], Value] | None = None,
load_state: Callable[[Iterator[Batch], Value], None] | None = None,
replay: bool = False,
place: int = unnamed(),
)

Records from factory(), a fresh iterator each call, read as one stream. Its position is what state(iterator) returns and load_state(iterator, saved) restores, as WebDataset, torchdata, Mosaic Streaming and Ray iterators each keep their own. replay declares instead that factory() reads the same records each time it opens, and a resume reads them again to its position (Replayed); a stream with neither trains, and its resume is refused. identity declares its content version in a run’s plan and its logs; with none, a resume needs resume="non-exact" (Manifest.exact).

def label() -> str
def describe() -> str
def fingerprint() -> Mapping[str, object]
def supports(read: Read) -> bool
def records(read: Read) -> Records
def manifest(version: int) -> ShardManifest
def metadata() -> Mapping[str, object]
def lengths() -> np.ndarray | None

function source

def from_iter(
factory: Callable[[], Iterator[Batch]],
*,
identity: str | None = None,
state: Callable[[Iterator[Batch]], Value] | None = None,
load_state: Callable[[Iterator[Batch], Value], None] | None = None,
replay: bool = False,
) -> Iterated

The records factory() iterates as a Dew dataset, its content version identity where given, resumed through the iterator’s own state and load_state where it has them, or by replay where it declares replay (Iterated).