Skip to content

dew.data.steps.windows

Overlapping causal windows, with the boundary target retained.

Grain’s concat-and-split packer has disjoint windows. Causal LM windows share their boundary token so every target is predicted exactly once.

NameSummary
TokenSpan
IndexedWindowsComplete causal windows of aligned fields backed by contiguous token spans.
WindowSourceCausal windows of length tokens, stride apart, over a token store’s tokens.
CausalWindowsWindows over a stream of documents; a mixture’s (documents) say, by id, which documents each holds and which end in it (DOCUMENT).

class source

class TokenSpan(Protocol)

dataclass source

class IndexedWindows(dtype: str | None = None)

Complete causal windows of aligned fields backed by contiguous token spans.

def from_records(rows: Sequence[Batch], windows: Windows) -> IndexedWindows

dataclass source

class WindowSource(rows: Rows = 'derived')

Causal windows of length tokens, stride apart, over a token store’s tokens.

max_length: int

The inputs each window holds, which a token budget divides into rows (dew.data.pipeline.TokenGeometry).

def describe() -> str
def fingerprint() -> Mapping[str, object]
def can_provision(read: Read) -> bool
def provision() -> WindowSource

Provision the token store, whose length index these windows need to read it by span.

def records(read: Read) -> IndexedWindows
def metadata() -> Mapping[str, object]
def lengths() -> np.ndarray | None
def manifest(version: int) -> ShardManifest

class source

class CausalWindows(
parent: grain.IterDataset[Batch],
windows: Windows,
documents: Documents | None = None,
)

Windows over a stream of documents; a mixture’s (documents) say, by id, which documents each holds and which end in it (DOCUMENT).