Skip to content

dew.data.steps.generic

Field selection, formatting and record filtering, steps any chain takes.

NameSummary
alignedelement’s fields, which hold one entry a token of the record and so one length, as arrays, beside its beside fields, on axes of their own and read as they are.
FieldsA record’s fields names (every field where None), renamed by rename, without remove.
CoordinatedA packed training or validation batch with each window’s documents as the model reads them: the <ids>_segment_ids and <ids>_positions a packer writes for its fields become <ids>_coordinates, the model’s dew.batch.Coordinates built on the host, holding at most max_segments documents a row, or as many as a window holds tokens where None (static, so a compiled step is specialized to it).
Formatting

function source

def aligned(
element: Batch,
fields: tuple[str, ...],
beside: tuple[str, ...] = (),
) -> Batch

element’s fields, which hold one entry a token of the record and so one length, as arrays, beside its beside fields, on axes of their own and read as they are. A record whose fields differ in length is refused: packed or cut into windows, each field would be cut at other places, one field’s tokens standing beside another’s from elsewhere, which no reader tells.

dataclass source

class Fields(
names: tuple[str, ...] | None = None,
rename: tuple[tuple[str, str], ...] = (),
remove: tuple[str, ...] = (),
bookkeeping: bool = False,
)

A record’s fields names (every field where None), renamed by rename, without remove. bookkeeping keeps Dew’s own record of the record (_dew_*) whatever the projection keeps, its counted targets among it (dew.data.steps.text.rebound), as a pipeline’s column steps do; a projection Dew reads for a packer keeps exactly the fields it names.

def declared(facts: Mapping[str, object]) -> Mapping[str, object]

The shapes of the fields kept, under their new names.

def map(element: Batch) -> Batch

dataclass source

class Coordinated()

A packed training or validation batch with each window’s documents as the model reads them: the <ids>_segment_ids and <ids>_positions a packer writes for its fields become <ids>_coordinates, the model’s dew.batch.Coordinates built on the host, holding at most max_segments documents a row, or as many as a window holds tokens where None (static, so a compiled step is specialized to it). A window holding more than its read declared is refused, which a sequence kernel would run into its neighbours’ sequences. ids is the packed ids field, the first of fields.

def map(element: Batch) -> Batch

dataclass source

class Formatting()
def map(element: Batch) -> Batch