Skip to content

dew.data.cache

Write-through artifacts with per-input completions and conditional publication.

NameSummary
AUTOMATICThe environment variable that names the directory a process’s runs cache their chains’ leading deterministic steps in (automatic_root), or turns that off (off).
automatic_rootWhere a run caches what its chains’ leading deterministic steps make of their records, with no cache= (dew.data.compile._automatic): $DEW_DATA_CACHE, else data in Dew’s cache directory (dew._cache.dew_cache_dir), a local root holding at most Cache.keep_local_bytes, its oldest generations evicted first (LocalRetention).
AbandonedAn upload or a seal the cache’s writer was closed before, which it counts (abandoned).
Transform
CachedValue
CacheStats
CacheCache a deterministic prefix without changing a filling run’s execution plan.

attribute source

AUTOMATIC = 'DEW_DATA_CACHE'

The environment variable that names the directory a process’s runs cache their chains’ leading deterministic steps in (automatic_root), or turns that off (off).

function source

def automatic_root(loading: Loading) -> str | None

Where a run caches what its chains’ leading deterministic steps make of their records, with no cache= (dew.data.compile._automatic): $DEW_DATA_CACHE, else data in Dew’s cache directory (dew._cache.dew_cache_dir), a local root holding at most Cache.keep_local_bytes, its oldest generations evicted first (LocalRetention). None where the run’s loading opts out (Loading(cache=False)) or the process does (DEW_DATA_CACHE=off).

class source

class Abandoned(RuntimeError)

An upload or a seal the cache’s writer was closed before, which it counts (abandoned).

attribute source

attribute source

class source

class CacheStats(TypedDict)

dataclass source

class Cache(
url: str,
spool_bytes: int | str = 256 * 2 ** 20,
mirror_bytes: int | str = 8 * 2 ** 30,
keep_local_bytes: int | str = 8 * 2 ** 30,
on_full: Literal['stop', 'block'] = 'stop',
upload_workers: int = 2,
retry: int = 3,
shard_bytes: int | str = 16 * 2 ** 20,
storage_options: Mapping[str, object] | None = None,
optional: bool = False,
evicts: bool = True,
)

Cache a deterministic prefix without changing a filling run’s execution plan.

A fill whose key a generation is published under (CURRENT) reads that generation’s completions in place of running its steps, so a later run reuses an earlier one’s complete fill. Otherwise each process keeps its own writes; it never discovers another filling writer’s rows, and a writer that completes every input of the source it is given publishes that source’s generation. Volatile prefixes run on every read and produce no completions. A shard a remote store took but reads back otherwise is read again up to retry more times (Store.put_immutable); a failed transfer raises after its transport’s own retries, and a write to this host’s disk or memory fails at once.

optional: bool

Whether a run reads on without it where its storage fails (no room, no permission, its index’s database full or read-only): a cache the compiler placed (dew.data.compile._automatic), which a run never needs, then looks nothing up, stops admitting and its records are computed, as they are with no cache (read_on). A cache a chain names stops the run.

evicts: bool

Whether a fill in a local root evicts its oldest generations to stay within keep_local_bytes (LocalRetention), or holds to the room the root has free and stops there: a stream’s automatic cache, whose corpus may be larger than the root, so a fill that cannot complete takes no generation another read keeps (row 4217).

def read_on(error: OSError | sqlite3.Error | ValueError | TypeError, doing: str) -> bool

Whether this cache reads on uncached after its storage raised error while doing: it is optional, and logs that it does.

def reference(manifest: DependencyManifest) -> str

The cache a plan reads, by the key of the rows it serves: where the cache lives is no part of the data, so a run resumes against the same rows cached anywhere.

def mapped(
parent: grain.MapDataset,
key: DependencyManifest,
*,
transform: Transform | None = None,
source: Source | None = None,
missing: Sequence[int] = (),
) -> _Mapped

The cached rows of parent, a source’s records by index: a published generation’s, or filled as they are read. A fill that completes every input publishes the generation of source, the source they are. The inputs missing that a read skips (a store’s absent records) are written as a fill starts, so it can complete without reading them.

def streamed(
reader: ShardOpener,
key: DependencyManifest,
*,
transform: Transform | None = None,
threads: int | None = None,
source: Source | None = None,
transparent: bool = False,
) -> ShardOpener

reader’s shards read through the cache: a published generation’s records read back, or each shard’s fill written as it is read, source’s generation published by the last of its shards’ fills to seal (row 4211). A transparent read saves each shard’s position as its reader does, so a position saved with the cache resumes without it and the reverse (row 4217).

def generation(key: DependencyManifest) -> Generation | None
def publish(
key: DependencyManifest,
source: Source,
*,
plan: Callable[[Generation], Mapping[str, np.ndarray]] | None = None,
) -> Generation

The generation of source under key, once its sealed fills cover every input: the one published already, unless plan adds a packing plan, which a new generation carries.