Skip to content

dew.data.sinks

Immutable exports to filesystem URLs: ArrayRecord shards Dew writes, and Parquet or any other format a native writer writes from one Arrow stream (Writer), all committed by one publisher.

NameSummary
write_array_recordpayloads written to a local ArrayRecord file at path, one record each in their order, the file closed whether or not they all were: the shard an export and a cache fill each upload.
WriterA native writer an export hands its rows to as one Arrow stream (pyarrow.RecordBatchReader, the stream Arrow’s dataset writer, Delta’s, Lance’s and DuckDB’s take, and a Spark session reads as a table): it writes them as files of format, one Files reads, under directory of filesystem, Arrow’s over the export’s storage.
passedOne pass over a store’s or a chain’s records, in their one order, as export writes them: a store as a chain of it, by a read it supports, physical shards or none.
exportWrite immutable shards, then conditionally commit their manifest and _SUCCESS.

function source

def write_array_record(path: Path, payloads: Iterable[bytes]) -> None

payloads written to a local ArrayRecord file at path, one record each in their order, the file closed whether or not they all were: the shard an export and a cache fill each upload.

class source

class Writer(Protocol)

A native writer an export hands its rows to as one Arrow stream (pyarrow.RecordBatchReader, the stream Arrow’s dataset writer, Delta’s, Lance’s and DuckDB’s take, and a Spark session reads as a table): it writes them as files of format, one Files reads, under directory of filesystem, Arrow’s over the export’s storage. The export then lists what it wrote and commits its manifest and _SUCCESS after them (commit_export), the one publisher every export shares, so a reader finds the whole export or, where the writer stopped, none. Files reads the files in the order of their paths.

def write(
rows: pa.RecordBatchReader,
directory: str,
filesystem: pafs.FileSystem,
) -> None

function source

def passed(values: Source | Dataset) -> Iterator[Batch]

One pass over a store’s or a chain’s records, in their one order, as export writes them: a store as a chain of it, by a read it supports, physical shards or none. Close it when done.

function source

def export(
values: Source | grain.IterDataset | Iterable[Batch | bytes | None] | Pipeline,
url: str,
*,
format: Literal['parquet', 'array_record'] | Writer = 'parquet',
rows_per_shard: int = 100000,
partitioning: tuple[str, ...] | None = None,
storage_options: Mapping[str, object] | None = None,
metadata: Mapping[str, object] | None = None,
length_field: str | None = None,
) -> Mapping[str, object]

Write immutable shards, then conditionally commit their manifest and _SUCCESS.

ArrayRecord shards are Dew’s own, one of rows_per_shard records each. Parquet is Arrow’s writer (_Parquet), each partition column a Hive directory; any other Writer, a native one of the caller’s, writes the rows as one Arrow stream. Each export owns a UUID directory, so racing exporters cannot share shards, and commits only once its writer has written them all.