Skip to content

dew.data.storage

URL storage, named credentials, bounded transport caching and conditional puts.

NameSummary
PreconditionFailedThe object changed since the caller read its version.
SourceChangedAn object Dew’s mirror downloaded holding other bytes than those it was to hold: the size and digest a caller named (source_snapshot), or its store’s version of it as the download began.
STOREDThe version each remote store names an object it holds by and changes as it is replaced (Store.get_versioned), which a read of a range of it can be held to (dew.data.declared.ranged): an S3 object’s ETag, a GCS object’s generation.
ConditionalWritesAn fsspec filesystem that supplies its own conditional writes (Store).
ObjectInfo
object_infoThe size and SHA-256 of payload, bytes, or a file read from its start in bounded chunks.
matchingRefuse an object at url whose size and digest, found, are not what was committed for it.
commit_exportCommit the manifest of an export at root, its files already written there: its schema, format, record codec, rows, files and metadata, and what its producer certified of its rows (an export its count rules and partitioning; a directory of existing ArrayRecord shards nothing); then the success marker its digest names.
TransportStats
BoundedFileCacheA filecache with an LRU byte limit and cross-process reader pins.
mirror_metricsWhat this process’s mirrors (dewcache::) did, summed over every one it opened, as a run logs it beside its data metrics (data/mirror/..., row 4299): none where it opened none.
MIRROREDThe key under which a loading worker’s position carries its process’s mirror counters (dew.data.readers.WorkerMirrors): {"process": <its process>, "counts": mirror_metrics()}.
MIRRORSThe key under which a position that reads its loading workers’ positions without keeping them (an elastic read’s order, dew.data.elastic_order) carries their mirror counters, by process.
carried_mirrorsEach loading worker’s mirror counters a reader’s position carries (MIRRORED, MIRRORS), by the worker’s process, each its latest: cumulative, so a worker’s counters count once.
HeldA reader’s position as it holds it: JSON’s values, its records any mapping (a mix’s SharedMapping).
PositionTextA reader’s positions written as JSON (json.dumps’s text), each a value a JSON config carries (dew._records.json_value), its mirror counters emptied (MIRRORED, MIRRORS: a read’s metrics, not its place): each value a shared mapping holds (a mix’s members’ positions and counts, dew.data.shared.SharedMapping, each a snapshot never written once held) written once and its text written again while the next position holds that same value, so a mix of a thousand members writes the members a read moved, not every member’s position and counts each batch (row 4740: 163 ms and 2.3 MB a batch at 1,126 members, the prefetch thread holding the GIL).
content_versionA local file’s full SHA256, persisted in Dew’s cache until its metadata changes: one record a file, its metadata beside the checksum, replaced whole (write_atomically) when the file changes, so versioning one file never waits on another’s, in this process or any other that shares the cache, and a file rewritten keeps one record.
pinned_by_urlWhether the object at path is one version by its URL alone: a Hub repository’s file at a full commit, which no write moves.
backingThe store fs reads from, through any transport cache over it.
localWhether fs is this host’s own disk, whose paths are where it mounted a file.
on_gcsWhether fs is Google Cloud Storage, which ArrayRecord’s and Arrow’s own readers open by URL.
in_memoryWhether fs is fsspec’s in-process memory store: neither this host’s disk nor the network.
Versioned
ReadableWhere an object’s bytes are read (Store.readable): the filesystem, the object’s path on it, and, where a copy is read (a Hub revision’s downloaded snapshot), the directory the copy’s own paths start at; the names above it are the copy’s location’s, not the object’s.
StoreAn object URL with compare-and-swap publication and verified uploads.

class source

class PreconditionFailed(FileExistsError)

The object changed since the caller read its version.

class source

class SourceChanged(OSError)

An object Dew’s mirror downloaded holding other bytes than those it was to hold: the size and digest a caller named (source_snapshot), or its store’s version of it as the download began.

attribute source

STORED = {'s3': 'etag', 'gs': 'generation', 'gcs': 'generation'}

The version each remote store names an object it holds by and changes as it is replaced (Store.get_versioned), which a read of a range of it can be held to (dew.data.declared.ranged): an S3 object’s ETag, a GCS object’s generation. A Hub file is held by its URL instead, at a commit (Store.pinned).

class source

class ConditionalWrites(Protocol)

An fsspec filesystem that supplies its own conditional writes (Store).

def put_conditional(path: str, payload: bytes, version: str | None) -> str

class source

class ObjectInfo(TypedDict)

function source

def object_info(payload: bytes | BinaryIO) -> ObjectInfo

The size and SHA-256 of payload, bytes, or a file read from its start in bounded chunks.

function source

def matching(found: ObjectInfo, expected: Mapping[str, object], url: str) -> None

Refuse an object at url whose size and digest, found, are not what was committed for it.

function source

def commit_export(
root: Store,
*,
format: str,
codec: int | None,
rows: int,
files: list[Mapping[str, object]],
metadata: Mapping[str, object] | None = None,
certified: Mapping[str, JSON] | None = None,
) -> Mapping[str, object]

Commit the manifest of an export at root, its files already written there: its schema, format, record codec, rows, files and metadata, and what its producer certified of its rows (an export its count rules and partitioning; a directory of existing ArrayRecord shards nothing); then the success marker its digest names. An export (dew.data.sinks.export) and existing shards provisioned in place (Files.provision) both commit through it.

class source

class TransportStats(TypedDict)

class source

class BoundedFileCache(*, max_bytes: int | str = 8 * 2 ** 30, **kwargs={})

A filecache with an LRU byte limit and cross-process reader pins.

Downloads share the cache directory’s lock. Oversized files, and misses while every eviction candidate is pinned, stream directly from the store.

def cat_file(
path: str,
start: int | None = None,
end: int | None = None,
**kwargs={},
) -> bytes
def metrics() -> TransportStats

function source

def mirror_metrics() -> dict[str, float]

What this process’s mirrors (dewcache::) did, summed over every one it opened, as a run logs it beside its data metrics (data/mirror/..., row 4299): none where it opened none.

attribute source

MIRRORED = 'dew_mirror'

The key under which a loading worker’s position carries its process’s mirror counters (dew.data.readers.WorkerMirrors): {"process": <its process>, "counts": mirror_metrics()}.

attribute source

MIRRORS = 'dew_mirrors'

The key under which a position that reads its loading workers’ positions without keeping them (an elastic read’s order, dew.data.elastic_order) carries their mirror counters, by process.

function source

def carried_mirrors(
state: Mapping[str, object] | Sequence[object] | None,
) -> dict[str, dict[str, float]]

Each loading worker’s mirror counters a reader’s position carries (MIRRORED, MIRRORS), by the worker’s process, each its latest: cumulative, so a worker’s counters count once.

attribute source

A reader’s position as it holds it: JSON’s values, its records any mapping (a mix’s SharedMapping).

class source

class PositionText()

A reader’s positions written as JSON (json.dumps’s text), each a value a JSON config carries (dew._records.json_value), its mirror counters emptied (MIRRORED, MIRRORS: a read’s metrics, not its place): each value a shared mapping holds (a mix’s members’ positions and counts, dew.data.shared.SharedMapping, each a snapshot never written once held) written once and its text written again while the next position holds that same value, so a mix of a thousand members writes the members a read moved, not every member’s position and counts each batch (row 4740: 163 ms and 2.3 MB a batch at 1,126 members, the prefetch thread holding the GIL). One reader owns one, writing its positions on one thread.

function source

def content_version(fs: AbstractFileSystem, path: str, size: int) -> str

A local file’s full SHA256, persisted in Dew’s cache until its metadata changes: one record a file, its metadata beside the checksum, replaced whole (write_atomically) when the file changes, so versioning one file never waits on another’s, in this process or any other that shares the cache, and a file rewritten keeps one record.

function source

def pinned_by_url(fs: AbstractFileSystem, path: str) -> bool

Whether the object at path is one version by its URL alone: a Hub repository’s file at a full commit, which no write moves.

function source

def backing(fs: AbstractFileSystem) -> AbstractFileSystem

The store fs reads from, through any transport cache over it.

function source

def local(fs: AbstractFileSystem) -> bool

Whether fs is this host’s own disk, whose paths are where it mounted a file.

function source

def on_gcs(fs: AbstractFileSystem) -> bool

Whether fs is Google Cloud Storage, which ArrayRecord’s and Arrow’s own readers open by URL.

function source

def in_memory(fs: AbstractFileSystem) -> bool

Whether fs is fsspec’s in-process memory store: neither this host’s disk nor the network.

dataclass source

class Versioned(data: bytes, version: str)

class source

class Readable(NamedTuple)

Where an object’s bytes are read (Store.readable): the filesystem, the object’s path on it, and, where a copy is read (a Hub revision’s downloaded snapshot), the directory the copy’s own paths start at; the names above it are the copy’s location’s, not the object’s. None where the object is read in place.

dataclass source

class Store(
url: str,
storage_options: Mapping[str, object] | None = None,
retention: LocalRetention | None = None,
)

An object URL with compare-and-swap publication and verified uploads.

put_if_absent and put_if_match run natively on local files, memory, S3 and the S3-compatible endpoints that honour its conditional PUTs, GCS and Hugging Face Hub repositories. Any other fsspec filesystem can supply them (ConditionalWrites) with a method put_conditional(path, payload, version), which writes payload only while the object is absent (version None) or still at version, and returns the new version. It reports a failed precondition with any FileExistsError, so it needs nothing from Dew, and Store raises that again as PreconditionFailed. The filesystem’s ukey(path) names the same versions, and get_versioned reads it on both sides of the bytes.

remote: bool

Whether this object lives behind a network service, where a failed write may pass and a retry is worth making; this host’s disk and memory fail for reasons a retry keeps.

def resolve() -> tuple[AbstractFileSystem, str]
def readable() -> Readable

Where this object’s bytes are read: a Hub repository’s file at a commit is read from huggingface_hub’s own cache where that holds it, as huggingface_hub reads its files, so a downloaded revision reads offline (HF_HUB_OFFLINE) and without a request online; any other object, and a Hub file the cache lacks, where resolve places it.

def pinned() -> Store

This object at the commit its revision names now, where it is a Hub repository’s file at a branch or a tag, which a later commit moves (a file at a commit is one version by its URL, pinned_by_url); this object itself on any other store, and at a commit already.

def cached() -> Store
def child(path: str) -> Store
def describe() -> Mapping[str, object]
def read() -> bytes
def size() -> int
def write(payload: bytes) -> None

Write payload here whole, replacing what was. A local file is replaced, not rewritten in place: a rewrite truncates it first, and a reader that mapped it, a store’s document index say, faults (SIGBUS) on any page past the cut until the write ends. An object store replaces an object whole already.

def verified(payload: bytes) -> dict[str, JSON]

Write payload here where nothing is, and read it back as written: its size and digest, and on a remote store the version it holds it at (held_at), to which a read of it is held (dew.data.declared.Declared).

def held_at(written: ObjectInfo | None = None) -> dict[str, JSON]

This object’s size and digest, read in one request with the version its store holds it at (get_versioned, named by STORED, on a remote store), refused unless they are written’s where given.

def checked(committed: Mapping[str, object]) -> bytes

This object’s bytes, refused unless they are the size and digest committed states.

def put_immutable(payload: bytes, retries: int = 0) -> str

Create this object from payload, or accept the byte-identical object an earlier attempt created, and return the stored version.

Other bytes already at the URL are refused with a ValueError. An object this call created that reads back otherwise is read again, up to retries times, a longer wait before each, then an OSError: no transport checks what a write left. A failed transfer raises at once, after the transport’s own retries (s3fs’s, gcsfs’s, huggingface_hub’s).

def get_versioned() -> Versioned
def put_if_absent(payload: bytes) -> str
def put_if_match(payload: bytes, version: str) -> str