Skip to content

dew.data.sources.grain

A Grain dataset read as a Dew store.

NameSummary
GrainA Grain MapDataset, read by index or as one stream, or an IterDataset, read as one stream whose position Grain checkpoints.
from_graindataset as a Dew dataset, its content version identity where given (Grain).

dataclass source

class Grain(
dataset: grain.MapDataset | grain.IterDataset,
identity: str | None = None,
place: int = unnamed(),
)

A Grain MapDataset, read by index or as one stream, or an IterDataset, read as one stream whose position Grain checkpoints. Grain states no store of its own, so identity declares its content version in a run’s plan, a cache’s key and its logs; with none, a run reads it and a resume needs resume="non-exact", and no cache keys it (Manifest.exact).

def label() -> str
def describe() -> str
def fingerprint() -> Mapping[str, object]
def supports(read: Read) -> bool
def records(read: Read) -> Records
def manifest(version: int) -> ShardManifest
def metadata() -> Mapping[str, object]
def lengths() -> np.ndarray | None

function source

def from_grain(
dataset: grain.MapDataset | grain.IterDataset,
identity: str | None = None,
) -> Grain

dataset as a Dew dataset, its content version identity where given (Grain).