dew.data.storage
URL storage, named credentials, bounded transport caching and conditional puts.
| Name | Summary |
|---|---|
PreconditionFailed | The object changed since the caller read its version. |
SourceChanged | An object Dew’s mirror downloaded holding other bytes than those it was to hold: the size and digest a caller named (source_snapshot), or its store’s version of it as the download began. |
STORED | The version each remote store names an object it holds by and changes as it is replaced (Store.get_versioned), which a read of a range of it can be held to (dew.data.declared.ranged): an S3 object’s ETag, a GCS object’s generation. |
ConditionalWrites | An fsspec filesystem that supplies its own conditional writes (Store). |
ObjectInfo | |
object_info | The size and SHA-256 of payload, bytes, or a file read from its start in bounded chunks. |
matching | Refuse an object at url whose size and digest, found, are not what was committed for it. |
commit_export | Commit the manifest of an export at root, its files already written there: its schema, format, record codec, rows, files and metadata, and what its producer certified of its rows (an export its count rules and partitioning; a directory of existing ArrayRecord shards nothing); then the success marker its digest names. |
TransportStats | |
BoundedFileCache | A filecache with an LRU byte limit and cross-process reader pins. |
mirror_metrics | What this process’s mirrors (dewcache::) did, summed over every one it opened, as a run logs it beside its data metrics (data/mirror/..., row 4299): none where it opened none. |
MIRRORED | The key under which a loading worker’s position carries its process’s mirror counters (dew.data.readers.WorkerMirrors): {"process": <its process>, "counts": mirror_metrics()}. |
MIRRORS | The key under which a position that reads its loading workers’ positions without keeping them (an elastic read’s order, dew.data.elastic_order) carries their mirror counters, by process. |
carried_mirrors | Each loading worker’s mirror counters a reader’s position carries (MIRRORED, MIRRORS), by the worker’s process, each its latest: cumulative, so a worker’s counters count once. |
Held | A reader’s position as it holds it: JSON’s values, its records any mapping (a mix’s SharedMapping). |
PositionText | A reader’s positions written as JSON (json.dumps’s text), each a value a JSON config carries (dew._records.json_value), its mirror counters emptied (MIRRORED, MIRRORS: a read’s metrics, not its place): each value a shared mapping holds (a mix’s members’ positions and counts, dew.data.shared.SharedMapping, each a snapshot never written once held) written once and its text written again while the next position holds that same value, so a mix of a thousand members writes the members a read moved, not every member’s position and counts each batch (row 4740: 163 ms and 2.3 MB a batch at 1,126 members, the prefetch thread holding the GIL). |
content_version | A local file’s full SHA256, persisted in Dew’s cache until its metadata changes: one record a file, its metadata beside the checksum, replaced whole (write_atomically) when the file changes, so versioning one file never waits on another’s, in this process or any other that shares the cache, and a file rewritten keeps one record. |
pinned_by_url | Whether the object at path is one version by its URL alone: a Hub repository’s file at a full commit, which no write moves. |
backing | The store fs reads from, through any transport cache over it. |
local | Whether fs is this host’s own disk, whose paths are where it mounted a file. |
on_gcs | Whether fs is Google Cloud Storage, which ArrayRecord’s and Arrow’s own readers open by URL. |
in_memory | Whether fs is fsspec’s in-process memory store: neither this host’s disk nor the network. |
Versioned | |
Readable | Where an object’s bytes are read (Store.readable): the filesystem, the object’s path on it, and, where a copy is read (a Hub revision’s downloaded snapshot), the directory the copy’s own paths start at; the names above it are the copy’s location’s, not the object’s. |
Store | An object URL with compare-and-swap publication and verified uploads. |
PreconditionFailed
Section titled “PreconditionFailed”class PreconditionFailed(FileExistsError)The object changed since the caller read its version.
SourceChanged
Section titled “SourceChanged”class SourceChanged(OSError)An object Dew’s mirror downloaded holding other bytes than those it was to hold: the size and
digest a caller named (source_snapshot), or its store’s version of it as the download began.
STORED
Section titled “STORED”STORED = {'s3': 'etag', 'gs': 'generation', 'gcs': 'generation'}The version each remote store names an object it holds by and changes as it is replaced
(Store.get_versioned), which a read of a range of it can be held to (dew.data.declared.ranged): an S3
object’s ETag, a GCS object’s generation. A Hub file is held by its URL instead, at a commit
(Store.pinned).
ConditionalWrites
Section titled “ConditionalWrites”class ConditionalWrites(Protocol)An fsspec filesystem that supplies its own conditional writes (Store).
ConditionalWrites.put_conditional
Section titled “ConditionalWrites.put_conditional”def put_conditional(path: str, payload: bytes, version: str | None) -> strObjectInfo
Section titled “ObjectInfo”class ObjectInfo(TypedDict)object_info
Section titled “object_info”def object_info(payload: bytes | BinaryIO) -> ObjectInfoThe size and SHA-256 of payload, bytes, or a file read from its start in bounded chunks.
matching
Section titled “matching”def matching(found: ObjectInfo, expected: Mapping[str, object], url: str) -> NoneRefuse an object at url whose size and digest, found, are not what was committed for it.
commit_export
Section titled “commit_export”def commit_export( root: Store, *, format: str, codec: int | None, rows: int, files: list[Mapping[str, object]], metadata: Mapping[str, object] | None = None, certified: Mapping[str, JSON] | None = None,) -> Mapping[str, object]Commit the manifest of an export at root, its files already written there: its schema,
format, record codec, rows, files and metadata, and what its producer certified of its rows (an
export its count rules and partitioning; a directory of existing ArrayRecord shards nothing);
then the success marker its digest names. An export (dew.data.sinks.export) and existing
shards provisioned in place (Files.provision) both commit through it.
TransportStats
Section titled “TransportStats”class TransportStats(TypedDict)BoundedFileCache
Section titled “BoundedFileCache”class BoundedFileCache(*, max_bytes: int | str = 8 * 2 ** 30, **kwargs={})A filecache with an LRU byte limit and cross-process reader pins.
Downloads share the cache directory’s lock. Oversized files, and misses while every eviction candidate is pinned, stream directly from the store.
BoundedFileCache.cat_file
Section titled “BoundedFileCache.cat_file”def cat_file( path: str, start: int | None = None, end: int | None = None, **kwargs={},) -> bytesBoundedFileCache.metrics
Section titled “BoundedFileCache.metrics”def metrics() -> TransportStatsmirror_metrics
Section titled “mirror_metrics”def mirror_metrics() -> dict[str, float]What this process’s mirrors (dewcache::) did, summed over every one it opened, as a run logs it
beside its data metrics (data/mirror/..., row 4299): none where it opened none.
MIRRORED
Section titled “MIRRORED”MIRRORED = 'dew_mirror'The key under which a loading worker’s position carries its process’s mirror counters
(dew.data.readers.WorkerMirrors): {"process": <its process>, "counts": mirror_metrics()}.
MIRRORS
Section titled “MIRRORS”MIRRORS = 'dew_mirrors'The key under which a position that reads its loading workers’ positions without keeping them (an
elastic read’s order, dew.data.elastic_order) carries their mirror counters, by process.
carried_mirrors
Section titled “carried_mirrors”def carried_mirrors( state: Mapping[str, object] | Sequence[object] | None,) -> dict[str, dict[str, float]]Each loading worker’s mirror counters a reader’s position carries (MIRRORED, MIRRORS), by the
worker’s process, each its latest: cumulative, so a worker’s counters count once.
A reader’s position as it holds it: JSON’s values, its records any mapping (a mix’s SharedMapping).
PositionText
Section titled “PositionText”class PositionText()A reader’s positions written as JSON (json.dumps’s text), each a value a JSON config carries
(dew._records.json_value), its mirror counters emptied (MIRRORED, MIRRORS: a read’s metrics, not its
place): each value a shared mapping holds (a mix’s
members’ positions and counts, dew.data.shared.SharedMapping, each a snapshot never written once held)
written once and its text written again while the next position holds that same value, so a mix of a
thousand members writes the members a read moved, not every member’s position and counts each batch
(row 4740: 163 ms and 2.3 MB a batch at 1,126 members, the prefetch thread holding the GIL). One reader
owns one, writing its positions on one thread.
content_version
Section titled “content_version”def content_version(fs: AbstractFileSystem, path: str, size: int) -> strA local file’s full SHA256, persisted in Dew’s cache until its metadata changes: one record a
file, its metadata beside the checksum, replaced whole (write_atomically) when the file changes,
so versioning one file never waits on another’s, in this process or any other that shares the
cache, and a file rewritten keeps one record.
pinned_by_url
Section titled “pinned_by_url”def pinned_by_url(fs: AbstractFileSystem, path: str) -> boolWhether the object at path is one version by its URL alone: a Hub repository’s file at a full
commit, which no write moves.
backing
Section titled “backing”def backing(fs: AbstractFileSystem) -> AbstractFileSystemThe store fs reads from, through any transport cache over it.
def local(fs: AbstractFileSystem) -> boolWhether fs is this host’s own disk, whose paths are where it mounted a file.
on_gcs
Section titled “on_gcs”def on_gcs(fs: AbstractFileSystem) -> boolWhether fs is Google Cloud Storage, which ArrayRecord’s and Arrow’s own readers open by URL.
in_memory
Section titled “in_memory”def in_memory(fs: AbstractFileSystem) -> boolWhether fs is fsspec’s in-process memory store: neither this host’s disk nor the network.
Versioned
Section titled “Versioned”class Versioned(data: bytes, version: str)Readable
Section titled “Readable”class Readable(NamedTuple)Where an object’s bytes are read (Store.readable): the filesystem,
the object’s path on it, and, where a copy is read (a Hub revision’s
downloaded snapshot), the directory the copy’s own paths start at; the
names above it are the copy’s location’s, not the object’s. None where
the object is read in place.
class Store( url: str, storage_options: Mapping[str, object] | None = None, retention: LocalRetention | None = None,)An object URL with compare-and-swap publication and verified uploads.
put_if_absent and put_if_match run natively on local files, memory,
S3 and the S3-compatible endpoints that honour its conditional PUTs, GCS
and Hugging Face Hub repositories. Any other fsspec filesystem can
supply them (ConditionalWrites) with a method put_conditional(path, payload, version), which writes payload only while the object is
absent (version None) or still at version, and returns the new
version. It reports a failed precondition with any FileExistsError, so
it needs nothing from Dew, and Store raises that again as
PreconditionFailed. The filesystem’s ukey(path) names the same
versions, and get_versioned reads it on both sides of the bytes.
remote: bool-
Whether this object lives behind a network service, where a failed write may pass and a retry is worth making; this host’s disk and memory fail for reasons a retry keeps.
Store.resolve
Section titled “Store.resolve”def resolve() -> tuple[AbstractFileSystem, str]Store.readable
Section titled “Store.readable”def readable() -> ReadableWhere this object’s bytes are read: a Hub repository’s file at a
commit is read from huggingface_hub’s own cache where that holds it, as
huggingface_hub reads its files, so a downloaded revision reads offline
(HF_HUB_OFFLINE) and without a request online; any other object, and a
Hub file the cache lacks, where resolve places it.
Store.pinned
Section titled “Store.pinned”def pinned() -> StoreThis object at the commit its revision names now, where it is a Hub repository’s file at a
branch or a tag, which a later commit moves (a file at a commit is one version by its URL,
pinned_by_url); this object itself on any other store, and at a commit already.
Store.cached
Section titled “Store.cached”def cached() -> StoreStore.child
Section titled “Store.child”def child(path: str) -> StoreStore.describe
Section titled “Store.describe”def describe() -> Mapping[str, object]Store.read
Section titled “Store.read”def read() -> bytesStore.size
Section titled “Store.size”def size() -> intStore.write
Section titled “Store.write”def write(payload: bytes) -> NoneWrite payload here whole, replacing what was. A local file is replaced, not
rewritten in place: a rewrite truncates it first, and a reader that mapped it, a
store’s document index say, faults (SIGBUS) on any page past the cut until the
write ends. An object store replaces an object whole already.
Store.verified
Section titled “Store.verified”def verified(payload: bytes) -> dict[str, JSON]Write payload here where nothing is, and read it back as written: its size and digest, and
on a remote store the version it holds it at (held_at), to which a read of it is held
(dew.data.declared.Declared).
Store.held_at
Section titled “Store.held_at”def held_at(written: ObjectInfo | None = None) -> dict[str, JSON]This object’s size and digest, read in one request with the version its store holds it at
(get_versioned, named by STORED, on a remote store), refused unless they are written’s
where given.
Store.checked
Section titled “Store.checked”def checked(committed: Mapping[str, object]) -> bytesThis object’s bytes, refused unless they are the size and digest committed states.
Store.put_immutable
Section titled “Store.put_immutable”def put_immutable(payload: bytes, retries: int = 0) -> strCreate this object from payload, or accept the byte-identical
object an earlier attempt created, and return the stored version.
Other bytes already at the URL are refused with a ValueError. An object
this call created that reads back otherwise is read again, up to
retries times, a longer wait before each, then an OSError: no
transport checks what a write left. A failed transfer raises at once,
after the transport’s own retries (s3fs’s, gcsfs’s, huggingface_hub’s).
Store.get_versioned
Section titled “Store.get_versioned”def get_versioned() -> VersionedStore.put_if_absent
Section titled “Store.put_if_absent”def put_if_absent(payload: bytes) -> strStore.put_if_match
Section titled “Store.put_if_match”def put_if_match(payload: bytes, version: str) -> str