Skip to content

dew.data.steps.chat

Block-diffusion response rows of rendered conversations, and prompt record encodings.

NameSummary
ResponseCanvasesOne block-diffusion row per assistant turn of a rendered conversation.
prompt_factsThe facts of rendered prompts: max_prompt_len ids and their length, and the reward fields reward_bytes wide where known, in place of the prompt’s columns.
RenderPrompt
ConfigurePrompt

dataclass source

class ResponseCanvases(pad_id: int = 0)

One block-diffusion row per assistant turn of a rendered conversation.

The row’s text is everything before the turn, pad_id-padded to prompt_length, then the turn’s completion over count canvases of canvas_size: eos_id fills the rest of its last canvas, pad_id the canvases after it, and canvas_mask marks the filled canvases, as the reference’s CanvasChunker lays a response out, and prompt_mask the context’s own ids, so no id the vocabulary also pads with reads as padding (gemma bf0b49901, diffusion/hackable_diffusion_adapter/data/data.py). A longer prompt or completion keeps its first ids, as the reference’s Pad(truncate=True) and CanvasChunker cut them (gm/data/_functional.py:83). Either cut marks the row truncated.

def declared(facts: Mapping[str, object]) -> Mapping[str, object]

A row for each assistant turn: its context and completion canvases as text, the canvases filled as canvas_mask and the context’s own ids as prompt_mask.

def flat_map(element: Batch) -> Sequence[Batch]

function source

def prompt_facts(
facts: Mapping[str, object],
tokenizer: str,
max_prompt_len: int,
reward_bytes: int | None,
) -> Mapping[str, object]

The facts of rendered prompts: max_prompt_len ids and their length, and the reward fields reward_bytes wide where known, in place of the prompt’s columns.

dataclass source

class RenderPrompt(
max_prompt_len: int = PROMPT_LEN,
pad_id: int = PROMPT_PAD,
reward_bytes: int = 1024,
long: Literal['drop', 'truncate'] = PROMPT_LONG,
thinking: bool | None = None,
)
vocabulary: ByteTokenizer | HFTokenizer

The tokenizer and its template, resolved once as Tokenize.vocabulary.

def dependencies() -> Mapping[str, object]
def declared(facts: Mapping[str, object]) -> Mapping[str, object]
def map(element: Batch) -> Batch | Rejected

dataclass source

class ConfigurePrompt()
def declared(facts: Mapping[str, object]) -> Mapping[str, object]

What the step it binds makes: its reward width the facts state, else measured.