dew.data.steps.text
Over-length policies, fixed-length checks and target counts of token records.
| Name | Summary |
|---|---|
LengthPolicy | |
ValidTargets | |
CountTargets | Each record’s targets the loss scores, counted once and carried on the record with the rule that counted them (recounted), so a token mixture weighs it without counting it again. |
rebound | Where a projection of a counted element (Fields) leaves the fields its count was counted from: renamed where it renames one, empty where it drops one, which no step can count from again; nothing for a record no count rides on. |
targets_changed | Whether a step that made output of given changed what its targets are counted from: a field the count reads that given held and output holds as a different array, or a counted field that output holds and given did not, as a map adding a row’s blocks does. |
recounted | output with the targets counted on given (CountTargets): carried onto a record the step made anew without them, with the rule and the names they were counted by; kept while the step left every field they were counted from as it was, as a map of a record’s other fields or a projection does; and counted again by the rule that counted them where it changed one in place, as a crop does, from the fields as they are named now. |
CountRule | The rule a record’s targets were counted by (CountTargets) and the name its roles go by (rebound): a windows or pack stage counts the windows it makes of such records by it, as the loss scores them, since their documents’ counts describe none of them. |
count_rules | The rules other than every target’s that rows’ targets were counted by (CountRule), as a store states them (count_rules): each a role and a pad id. |
stated_rules | The rules a store’s count_rules states, each a role and a pad id (CountRule), either None, as a generation writes them. |
counted_window | A window a pack or windows stage made, its targets of its token field counted as the loss scores them and carried on it, by the rule its documents were counted by (CountRule), or every target’s where nothing counted them: a later step, the read that accepts it (a memory read’s, dew.data.lowering._Build.targeted) and a mixture by tokens read them as a counted record’s. |
LayoutRule | The one rule the documents a pack or windows stage lays out were counted by (CountRule), or None where nothing counted them, as each document passes. |
FixedTokens | Rows of exactly max_length + extra tokens of field, the max_length a token budget divides into rows (TokenGeometry). |
LengthPolicy
Section titled “LengthPolicy”class LengthPolicy( field: str = 'text', long: Literal['truncate', 'drop'] = 'truncate', side: Literal['left', 'right'] = 'right', aligned: tuple[str, ...] = (),)rejects: bool-
Whether a long record is dropped (
filter_map) rather than cut (map).
LengthPolicy.map
Section titled “LengthPolicy.map”def map(element: Batch) -> Batch | RejectedValidTargets
Section titled “ValidTargets”class ValidTargets( field: str = 'text', role: int | None = None, pad_id: int | None = None, measured: bool = True,)CountTargets
Section titled “CountTargets”class CountTargets(role: int | None = None, pad_id: int | None = None)Each record’s targets the loss scores, counted once and carried on the record with the rule
that counted them (recounted), so a token mixture weighs it without counting it again.
CountTargets.declared
Section titled “CountTargets.declared”def declared(facts: Mapping[str, object]) -> Mapping[str, object]The records as they were, their targets counted.
CountTargets.map
Section titled “CountTargets.map”def map(element: Batch) -> Batch | Rejectedrebound
Section titled “rebound”def rebound(element: Batch, kept: Batch, renamed: Mapping[str, str]) -> BatchWhere a projection of a counted element (Fields) leaves the fields its count was counted
from: renamed where it renames one, empty where it drops one, which no step can count from
again; nothing for a record no count rides on.
targets_changed
Section titled “targets_changed”def targets_changed(given: Batch, output: Batch) -> boolWhether a step that made output of given changed what its targets are counted from: a field the
count reads that given held and output holds as a different array, or a counted field that output
holds and given did not, as a map adding a row’s blocks does. Each is read where the record names it
before the step and where output names it after (rebound), so a projection that moves a field, or
names another field as it was, changes nothing; a field the step removed pins the count, which nothing it
still holds could give again.
recounted
Section titled “recounted”def recounted(given: Batch, output: Batch) -> Batchoutput with the targets counted on given (CountTargets): carried onto a record the step
made anew without them, with the rule and the names they were counted by; kept while the step
left every field they were counted from as it was, as a map of a record’s other fields or a
projection does; and counted again by the rule that counted them where it changed one in place,
as a crop does, from the fields as they are named now. Every read carries a count so, a memory
read’s among them, which keeps no count of its own beside its records.
CountRule
Section titled “CountRule”class CountRule()The rule a record’s targets were counted by (CountTargets) and the name its roles go by
(rebound): a windows or pack stage counts the windows it makes of such records by it, as the
loss scores them, since their documents’ counts describe none of them.
CountRule.of
Section titled “CountRule.of”def of(element: Batch) -> CountRule | NoneThe rule element’s targets were counted by, or None where it is every target’s, as an
uncounted row’s are (ValidTargets): no role and no padding.
CountRule.counted
Section titled “CountRule.counted”def counted(window: Batch, field: str) -> Batchwindow, made of records counted by this rule from its token field, its own targets
counted by the rule and carrying it, so a later step or layout counts them by it again.
count_rules
Section titled “count_rules”def count_rules(rows: Iterable[Batch | bytes | None]) -> list[list[int | None]]The rules other than every target’s that rows’ targets were counted by (CountRule), as
a store states them (count_rules): each a role and a pad id.
stated_rules
Section titled “stated_rules”def stated_rules(value: object) -> set[tuple[int | None, int | None]]The rules a store’s count_rules states, each a role and a pad id (CountRule), either
None, as a generation writes them.
counted_window
Section titled “counted_window”def counted_window(window: Batch, rule: CountRule | None, field: str) -> BatchA window a pack or windows stage made, its targets of its token field counted as the loss
scores them and carried on it, by the rule its documents were counted by (CountRule), or every
target’s where nothing counted them: a later step, the read that accepts it (a memory read’s,
dew.data.lowering._Build.targeted) and a mixture by tokens read them as a counted record’s.
LayoutRule
Section titled “LayoutRule”class LayoutRule()The one rule the documents a pack or windows stage lays out were counted by (CountRule),
or None where nothing counted them, as each document passes. Documents counted by two rules,
or some by none, make windows no one rule counts, and are refused.
LayoutRule.of
Section titled “LayoutRule.of”def of(documents: Iterable[Batch]) -> CountRule | NoneThe one rule documents were counted by, all of them passing.
LayoutRule.state
Section titled “LayoutRule.state”def state() -> Mapping[str, object]LayoutRule.restore
Section titled “LayoutRule.restore”def restore(state: Mapping[str, object]) -> NoneFixedTokens
Section titled “FixedTokens”class FixedTokens(extra: int = 0, field: str = 'text')Rows of exactly max_length + extra tokens of field, the
max_length a token budget divides into rows (TokenGeometry).
FixedTokens.map
Section titled “FixedTokens.map”def map(element: Batch) -> Batch