Skip to content

dew.config.lm

The language-model run, as one typed record.

LMRunConfig is what an LM run parses from its command line and writes as run.json next to the checkpoints. Every run records the model, the data, the optimizer and the trainer. A decoder run also records the tokenizer its ids came from and the preview policy it generates with. TextGeneration.from_run and Pretrained.from_run read those two fields back from that file, so a run that does not record them loads as weights with no way to turn text into ids.

The three objectives a decoder can be trained under are one field, because they share every other field. lm is the next-token loss, masked_diffusion is MDLM over a bidirectional model, and block_diffusion is DiffusionGemma’s own fine-tuning objective. run trains it on the token files dew tokenize wrote, which decide the vocabulary.

NameSummary
LMRunConfigA decoder’s run (DecoderRunConfig) plus the settings specific to language models.

dataclass source

class LMRunConfig(
model: ModelConfig = (lambda: ModelConfig('causal_transformer', {}))(),
data: Annotated[Dataset | None, tyro.conf.Suppress] = None,
validation: Annotated[Dataset | Mapping[str, Dataset] | None, tyro.conf.Suppress] = None,
data_seed: int = 0,
data_plan_version: int | None = None,
source: SourceOptions = SourceOptions(),
optim: OptimConfig = (lambda: OptimConfig(learning_rate=0.0006, weight_decay=0.1, clip_grads=1.0))(),
trainer: TrainerConfig = TrainerConfig(),
objective: ObjectiveConfig = (lambda: ObjectiveConfig('lm'))(),
lora: Annotated[LoRA, tyro.conf.subcommand(lora)] | None = None,
val_metrics: Annotated[tuple[MetricConfig, ...], record_argument(tuple[MetricConfig, ...])] | None = None,
val_every: Annotated[Mapping[str, int | str], record_argument(Mapping[str, int | str])] = dict(),
val_summaries: Annotated[tuple[Summary, ...], record_argument(tuple[Summary, ...])] = (),
loop_steps: int | None = None,
loop_layers: tuple[int, int] | None = None,
loop_backprop_steps: int | None = None,
loop_remat: bool = False,
tokenizer: str = 'byte',
prompt: str = '',
max_new_tokens: int = 128,
sampling: Sampling = (lambda: Sampling(temperature=0.8, top_k=40))(),
pretrained: str | None = None,
rollout: Annotated[PromptRollout, tyro.conf.subcommand('rollout')] | None = None,
)

A decoder’s run (DecoderRunConfig) plus the settings specific to language models.

objective: ObjectiveConfig

The objective and its arguments: lm, masked_diffusion (MDLM), block_diffusion (the official DiffusionGemma fine-tuning objective), or any objective by its import path that LMObjective’s arguments build: the decoder, seq_len, its starting variables, samples and processor. A step whose optimizer clips by the attention logit maxima (muonclip) asks the objective for them (Step.qk_stats), whichever objective it is.

tokenizer: str

The tokenizer the ids were written with: ‘byte’, or an HF tokenizer name.

prompt: str

The prompt that validation samples continue; an empty prompt continues a newline.

max_new_tokens: int

The number of tokens generated per validation sample; 0 logs no text.

sampling: Sampling

The preview policy, recorded with the run for inference.

pretrained: str | None

The Hugging Face decoder to continue training from: a hub repo id, repo@revision (a branch, tag or commit), or a local directory in that layout. A run records a hub repo as repo@commit, with the commit it resolved to. The checkpoint sets the architecture, so of its fields —model.max-seq-len alone may be set.

rollout: Annotated[PromptRollout, tyro.conf.subcommand('rollout')] | None

The rollout a GRPO run trains on (--rollout.reward), over its chain’s prompts (dew.data.presets.prompts): GRPO needs one, and another objective takes none. The model’s context is a prompt and its completion.

def build_run() -> Prepared

The objective this run names over its token files.

The files decide the vocabulary; training and sampling decide the context, since generation decodes into a cache sized once at build time. A pretrained checkpoint decides the architecture instead, and the run records it as the commit it was read at. A fresh model records its vocabulary and context; a fine-tune records only its model choices, since the source supplies its geometry on every load. The objective’s inference record holds the built model for dew.pipeline. An adapter binds to the model and the tree training starts from, the loaded weights or a fresh draw from the run’s key, so the objective trains its factors alone. Each validation pass scores val_metrics, perplexity by default.

def default_metrics(*, validates: bool) -> tuple[MetricConfig, ...]

Perplexity, where the run evaluates a validation split it holds.

def tokens() -> Tokens

The token windows this run trains on, as its chain states them (Dataset.facts): their tokenizer, vocabulary and length. Raises ValueError for a chain that states none, which an LM run cannot size its model for.

def processor()

The run’s tokenizer as its checkpoints record it for every loader.

def samples() -> Samples

What the objective generates and decodes at every validation. The policy is kept with no preview budget too: the run records it, and its export and dew.pipeline decode with it.

def pinned() -> LMRunConfig

This run with its Hub sources at their commits (RunConfig.pinned), pretrained among them (SourceReference.pinned).

def pretrained_source(written: str, vocab_size: int, context: int)

The bundle pretrained names, at the commit the run pinned it to (pinned).

The checkpoint decides every architecture field, so the model flags may say only how far the KV cache reaches, the compute dtype and the attention kernel. The token files’ tokenizer has to be the checkpoint’s: continuing pretraining on ids from another vocabulary trains the embedding table against noise.

def masked_objective(model, fields, variables, source=None)

MDLM over a bidirectional model, its mask id from the model’s fields.

A pretrained diffusion checkpoint carries mask_token_id; a from-scratch run names it beside —model.no-causal. A token window is seq_len + 1 ids wide, and masked diffusion has no shift, so it denoises the whole row. The validation text is the unmasked rows decoded with the run’s tokenizer, or bare ids with no preview budget. A loaded source is what the objective binds, so its end-of-sequence ids reach the task.

def block_objective(model, variables, source=None)

Each complete token-window row split into the clean prompt the run names (--objective.prompt-length) and response canvases, over a loaded source where there is one, whose end-of-sequence ids the task takes.