Skip to content

dew.data.sources.hub

Prepared HF Arrow tables and physical Parquet shards from HfFileSystem.

NameSummary
hold_progress_lockGive tqdm’s own progress-bar class a lock, before any call into datasets.
split_filesThe paths of a repository listing that datasets gives split: of a configuration its card states (config), by its data_files patterns, else by datasets’ default patterns (datasets.data_files.get_data_patterns), under its data_dir where it names one, as datasets reads a card’s configuration (datasets.load.create_builder_configs_from_metadata_configs); resolved as it resolves a split’s patterns (DataFilesList.from_patterns), over an empty in-memory copy of the listing: the listing itself, which the Hub answered once and offline replays, is all the patterns read.
conversion_layoutWhether a commit’s listing is laid out as the Hub’s Parquet conversion lays its files out (refs/convert/parquet): no card, and each file but its .gitattributes a Parquet shard in a directory of its split in a directory of its configuration (<config>/<split>/<shard>.parquet, partial-<split> for a split converted in part: https://huggingface.co/docs/dataset-viewer/parquet).
HubA pinned split of a Hub dataset: name is its configuration, as datasets.load_dataset(repo, name=) takes it, and columns the fields each record keeps (all where None).

function source

def hold_progress_lock() -> None

Give tqdm’s own progress-bar class a lock, before any call into datasets. datasets maps a split’s files through tqdm’s thread_map (resolving a listing’s files to a split, reading or preparing its shards), which lends the bar class it is given a lock for the map and deletes it after where neither the class nor a class it derives from held one. Two maps at once (compile’s readers: a holdout’s train and held; several streamed sources discovered together) then delete it under each other’s bars: “type object ‘tqdm’ has no attribute ‘_lock’”. Held by tqdm’s class, which every bar class datasets lends a map derives from, each map finds it and restores it instead.

function source

def split_files(
paths: Sequence[str],
split: str,
config: Mapping[str, object] | None = None,
roots: Sequence[str] = (),
) -> list[str]

The paths of a repository listing that datasets gives split: of a configuration its card states (config), by its data_files patterns, else by datasets’ default patterns (datasets.data_files.get_data_patterns), under its data_dir where it names one, as datasets reads a card’s configuration (datasets.load.create_builder_configs_from_metadata_configs); resolved as it resolves a split’s patterns (DataFilesList.from_patterns), over an empty in-memory copy of the listing: the listing itself, which the Hub answered once and offline replays, is all the patterns read. A pattern that names the listing by a URL of its root (roots) reads it there; one that names a file elsewhere leaves the split none of the listing’s.

function source

def conversion_layout(paths: Sequence[str]) -> bool

Whether a commit’s listing is laid out as the Hub’s Parquet conversion lays its files out (refs/convert/parquet): no card, and each file but its .gitattributes a Parquet shard in a directory of its split in a directory of its configuration (<config>/<split>/<shard>.parquet, partial-<split> for a split converted in part: https://huggingface.co/docs/dataset-viewer/parquet). A repository’s own files may be laid out so too; only the conversion’s ref says which commit is it.

dataclass source

class Hub(
repo: str,
name: str | None = None,
split: str = 'train',
columns: tuple[str, ...] | None = None,
revision: str | None = None,
cache_dir: str | None = None,
)

A pinned split of a Hub dataset: name is its configuration, as datasets.load_dataset(repo, name=) takes it, and columns the fields each record keeps (all where None). Read whole or by index from the Arrow cache it prepares, or streamed from the repo’s own shards.

commit: str

The commit revision names, the default branch’s where None (dew._cache.hub_commit): online the Hub’s answer, recorded; offline (HF_HUB_OFFLINE) the cache’s ref, else the answer recorded online.

builder

datasets’ builder of this split at commit. With the Hub offline, where commit is the answer recorded online, datasets cannot rebuild a Hub dataset’s builder, so it is datasets’ own builder of the cache it prepared the split in at that commit (Cache), at the configuration and version the online builder recorded, and a split never prepared raises.

streamed: str | None

The pinned commit with readable shards of this split, or None. Native shards come from the resolved commit. An unpinned default revision without native shards can read its Parquet conversion.

def with_split(name: str) -> Hub
def label() -> str
def describe() -> str
def consumed(read: Read) -> Source
def fingerprint() -> Mapping[str, object]

The pinned split Arrow is prepared from. Its identity is the same on any host, in any cache directory, whether or not Arrow was prepared there. A streamed read consumes files, whose snapshot names its physical shards at their own pinned commit.

def supports(read: Read) -> bool
def can_provision(read: Read) -> bool
def preparation_bytes() -> int | None

What datasets downloads to prepare the split, as its builder’s info states it.

def provision() -> Hub
def records(read: Read) -> Records

Each row as the split stores it: an image column’s encoded bytes, which the image steps decode, so a memory read holds the split, not its decoded pixels.

def manifest(version: int) -> ShardManifest
def stated() -> Mapping[str, object]

Its builder’s info, which its metadata is, no row read; a split never prepared, with the Hub offline, states nothing.

def metadata() -> Mapping[str, object]
def lengths() -> np.ndarray | None