Skip to content

dew.data.sources.hf

A Hugging Face datasets dataset read as a Dew store.

NameSummary
HFA datasets.Dataset, read by index from its Arrow table or as one stream of it, or an IterableDataset, read over its shards whose positions its own state_dict checkpoints; the .map()s it holds are already applied.
arrow_rowsAn Arrow dataset’s rows by index, as Dew reads them: its image columns undecoded, which Dew’s own steps decode, and each row a record (_record).
class_namesThe names of each ClassLabel feature’s classes, by feature, which a configured caption reads in place of the label’s number.
from_hfA Hugging Face datasets.Dataset or IterableDataset as a Dew dataset (HF): an IterableDataset with its identity, unshuffled.

dataclass source

class HF(
dataset: datasets.Dataset | datasets.IterableDataset,
identity: str | None = None,
)

A datasets.Dataset, read by index from its Arrow table or as one stream of it, or an IterableDataset, read over its shards whose positions its own state_dict checkpoints; the .map()s it holds are already applied. An image column reads as its encoded bytes, which images.DecodeImage decodes. identity is its content version in a run’s plan, a cache’s key and its logs, its fingerprint by default; a stream has no fingerprint of its content, so it is named, with its version, or refused.

A stream resumes exactly from its state_dict only unshuffled, since Hugging Face’s shuffle buffer is not part of that state: one that shuffles through it is refused; shuffle with the chain’s shuffle(window=) instead.

A pool larger than the physical shards uses a row-wise node split. These parts resume exactly at the same topology, but their cache fill cannot publish: publication needs the store’s own shards.

def label() -> str
def describe() -> str
def fingerprint() -> Mapping[str, object]
def supports(read: Read) -> bool
def records(read: Read) -> Records
def manifest(version: int) -> ShardManifest
def metadata() -> Mapping[str, object]

Its class names (class_names), as Hub states a prepared dataset’s, and its rows where the dataset is indexed.

def lengths() -> np.ndarray | None

function source

def arrow_rows(dataset: datasets.Dataset) -> grain.MapDataset

An Arrow dataset’s rows by index, as Dew reads them: its image columns undecoded, which Dew’s own steps decode, and each row a record (_record). A Hugging Face dataset and a prepared Hub split read their rows through it.

function source

def class_names(features: Mapping[str, object] | None) -> dict[str, tuple[str, ...]]

The names of each ClassLabel feature’s classes, by feature, which a configured caption reads in place of the label’s number.

function source

def from_hf(
dataset: datasets.Dataset | datasets.IterableDataset,
identity: str | None = None,
) -> HF

A Hugging Face datasets.Dataset or IterableDataset as a Dew dataset (HF): an IterableDataset with its identity, unshuffled.