Skip to content

dew.data.sources.memory

Sources whose records are already resident in the caller’s memory.

NameSummary
DataFrameA table that hands itself over through the dataframe interchange protocol: a pandas or polars DataFrame, or an Arrow table.
SparkFrame
MemoryRecords, column arrays, or a DataFrame or Arrow table (DataFrame), with immutable positional IDs.

class source

class DataFrame(Protocol)

A table that hands itself over through the dataframe interchange protocol: a pandas or polars DataFrame, or an Arrow table.

class source

class SparkFrame(Protocol)
def toArrow() -> DataFrame

dataclass source

class Memory(
values: Records | Sequence[Batch] | Mapping[str, Sequence] | DataFrame,
identity: str | None = None,
shard_rows: int = 1024,
)

Records, column arrays, or a DataFrame or Arrow table (DataFrame), with immutable positional IDs. A table’s rows become records of its columns, its index none of them.

def from_spark(frame: SparkFrame) -> Memory

Gather a small pyspark 4.0+ frame through its Arrow handoff.

def label() -> str
def describe() -> str
def fingerprint() -> Mapping[str, object]

Its content, hashed once a compilation (dew.data.plan.once), which a plan, a cache key and a description each read, its records the caller’s to change between two.

def supports(read: Read) -> bool
def records(read: Read) -> Records
def manifest(version: int) -> ShardManifest
def metadata() -> Mapping[str, object]
def lengths() -> np.ndarray | None