dew.data.sources.hf
A Hugging Face datasets dataset read as a Dew store.
| Name | Summary |
|---|---|
HF | A datasets.Dataset, read by index from its Arrow table or as one stream of it, or an IterableDataset, read over its shards whose positions its own state_dict checkpoints; the .map()s it holds are already applied. |
arrow_rows | An Arrow dataset’s rows by index, as Dew reads them: its image columns undecoded, which Dew’s own steps decode, and each row a record (_record). |
class_names | The names of each ClassLabel feature’s classes, by feature, which a configured caption reads in place of the label’s number. |
from_hf | A Hugging Face datasets.Dataset or IterableDataset as a Dew dataset (HF): an IterableDataset with its identity, unshuffled. |
class HF( dataset: datasets.Dataset | datasets.IterableDataset, identity: str | None = None,)A datasets.Dataset, read by index from its Arrow table or as one
stream of it, or an IterableDataset, read over its shards whose
positions its own state_dict checkpoints; the .map()s it holds are already applied. An
image column reads as its encoded bytes, which images.DecodeImage decodes.
identity is its content version in a run’s plan, a cache’s key and its logs, its
fingerprint by default; a stream has no fingerprint of its content, so
it is named, with its version, or refused.
A stream resumes exactly from its state_dict only unshuffled, since
Hugging Face’s shuffle buffer is not part of that state: one that
shuffles through it is refused; shuffle with the chain’s
shuffle(window=) instead.
A pool larger than the physical shards uses a row-wise node split. These parts resume exactly at the same topology, but their cache fill cannot publish: publication needs the store’s own shards.
HF.label
Section titled “HF.label”def label() -> strHF.describe
Section titled “HF.describe”def describe() -> strHF.fingerprint
Section titled “HF.fingerprint”def fingerprint() -> Mapping[str, object]HF.supports
Section titled “HF.supports”def supports(read: Read) -> boolHF.records
Section titled “HF.records”def records(read: Read) -> RecordsHF.manifest
Section titled “HF.manifest”def manifest(version: int) -> ShardManifestHF.metadata
Section titled “HF.metadata”def metadata() -> Mapping[str, object]Its class names (class_names), as Hub states a prepared dataset’s, and
its rows where the dataset is indexed.
HF.lengths
Section titled “HF.lengths”def lengths() -> np.ndarray | Nonearrow_rows
Section titled “arrow_rows”def arrow_rows(dataset: datasets.Dataset) -> grain.MapDatasetAn Arrow dataset’s rows by index, as Dew reads them: its image columns undecoded, which Dew’s
own steps decode, and each row a record (_record). A Hugging Face dataset and a prepared Hub
split read their rows through it.
class_names
Section titled “class_names”def class_names(features: Mapping[str, object] | None) -> dict[str, tuple[str, ...]]The names of each ClassLabel feature’s classes, by feature, which a
configured caption reads in place of the label’s number.
from_hf
Section titled “from_hf”def from_hf( dataset: datasets.Dataset | datasets.IterableDataset, identity: str | None = None,) -> HFA Hugging Face datasets.Dataset or IterableDataset as a Dew
dataset (HF): an IterableDataset with its identity, unshuffled.