dew.data.cache
Write-through artifacts with per-input completions and conditional publication.
| Name | Summary |
|---|---|
AUTOMATIC | The environment variable that names the directory a process’s runs cache their chains’ leading deterministic steps in (automatic_root), or turns that off (off). |
automatic_root | Where a run caches what its chains’ leading deterministic steps make of their records, with no cache= (dew.data.compile._automatic): $DEW_DATA_CACHE, else data in Dew’s cache directory (dew._cache.dew_cache_dir), a local root holding at most Cache.keep_local_bytes, its oldest generations evicted first (LocalRetention). |
Abandoned | An upload or a seal the cache’s writer was closed before, which it counts (abandoned). |
Transform | |
CachedValue | |
CacheStats | |
Cache | Cache a deterministic prefix without changing a filling run’s execution plan. |
AUTOMATIC
Section titled “AUTOMATIC”AUTOMATIC = 'DEW_DATA_CACHE'The environment variable that names the directory a process’s runs cache their chains’ leading
deterministic steps in (automatic_root), or turns that off (off).
automatic_root
Section titled “automatic_root”def automatic_root(loading: Loading) -> str | NoneWhere a run caches what its chains’ leading deterministic steps make of their records, with no
cache= (dew.data.compile._automatic): $DEW_DATA_CACHE, else data in Dew’s cache directory
(dew._cache.dew_cache_dir), a local root holding at most Cache.keep_local_bytes, its oldest
generations evicted first (LocalRetention). None where the run’s loading opts out
(Loading(cache=False)) or the process does (DEW_DATA_CACHE=off).
Abandoned
Section titled “Abandoned”class Abandoned(RuntimeError)An upload or a seal the cache’s writer was closed before, which it counts (abandoned).
Transform
Section titled “Transform”CachedValue
Section titled “CachedValue”CacheStats
Section titled “CacheStats”class CacheStats(TypedDict)class Cache( url: str, spool_bytes: int | str = 256 * 2 ** 20, mirror_bytes: int | str = 8 * 2 ** 30, keep_local_bytes: int | str = 8 * 2 ** 30, on_full: Literal['stop', 'block'] = 'stop', upload_workers: int = 2, retry: int = 3, shard_bytes: int | str = 16 * 2 ** 20, storage_options: Mapping[str, object] | None = None, optional: bool = False, evicts: bool = True,)Cache a deterministic prefix without changing a filling run’s execution plan.
A fill whose key a generation is published under (CURRENT) reads that
generation’s completions in place of running its steps, so a later run
reuses an earlier one’s complete fill. Otherwise each process keeps its
own writes; it never discovers another filling writer’s rows, and a
writer that completes every input of the source it is given publishes
that source’s generation. Volatile prefixes run on every read and produce
no completions. A shard a remote store took but reads back otherwise is
read again up to retry more times (Store.put_immutable); a failed
transfer raises after its transport’s own retries, and a write to this
host’s disk or memory fails at once.
optional: bool-
Whether a run reads on without it where its storage fails (no room, no permission, its index’s database full or read-only): a cache the compiler placed (
dew.data.compile._automatic), which a run never needs, then looks nothing up, stops admitting and its records are computed, as they are with no cache (read_on). A cache a chain names stops the run. evicts: bool-
Whether a fill in a local root evicts its oldest generations to stay within
keep_local_bytes(LocalRetention), or holds to the room the root has free and stops there: a stream’s automatic cache, whose corpus may be larger than the root, so a fill that cannot complete takes no generation another read keeps (row 4217).
Cache.read_on
Section titled “Cache.read_on”def read_on(error: OSError | sqlite3.Error | ValueError | TypeError, doing: str) -> boolWhether this cache reads on uncached after its storage raised error while doing: it is
optional, and logs that it does.
Cache.reference
Section titled “Cache.reference”def reference(manifest: DependencyManifest) -> strThe cache a plan reads, by the key of the rows it serves: where the cache lives is no part of the data, so a run resumes against the same rows cached anywhere.
Cache.mapped
Section titled “Cache.mapped”def mapped( parent: grain.MapDataset, key: DependencyManifest, *, transform: Transform | None = None, source: Source | None = None, missing: Sequence[int] = (),) -> _MappedThe cached rows of parent, a source’s records by index: a published
generation’s, or filled as they are read. A fill that completes every
input publishes the generation of source, the source they are. The
inputs missing that a read skips (a store’s absent records) are written
as a fill starts, so it can complete without reading them.
Cache.streamed
Section titled “Cache.streamed”def streamed( reader: ShardOpener, key: DependencyManifest, *, transform: Transform | None = None, threads: int | None = None, source: Source | None = None, transparent: bool = False,) -> ShardOpenerreader’s shards read through the cache: a published generation’s records read back, or each
shard’s fill written as it is read, source’s generation published by the last of its shards’
fills to seal (row 4211). A transparent read saves each shard’s position as its reader does,
so a position saved with the cache resumes without it and the reverse (row 4217).
Cache.generation
Section titled “Cache.generation”def generation(key: DependencyManifest) -> Generation | NoneCache.publish
Section titled “Cache.publish”def publish( key: DependencyManifest, source: Source, *, plan: Callable[[Generation], Mapping[str, np.ndarray]] | None = None,) -> GenerationThe generation of source under key, once its sealed fills cover
every input: the one published already, unless plan adds a packing
plan, which a new generation carries.