Skip to content

dew.config

Typed configuration for a training run.

One dataclass tree describes a run: what to build, what to feed it, how to optimize it, and how the trainer runs. Recipes parse it with tyro, build the objective and the data it names, and pass both to RunConfig.train, so to_dict() is a full record of a run and from_dict() rebuilds it.

The model is its class and the fields the run gives it: --model hybrid_dit (an alias or an import path) picks the class, and each of its own fields is a flag typed by its annotation (--model.num_layers 12, --model.dtype bfloat16). The data is a chain (dew.data), which the record writes through the data’s codec (dew.data.records) and the command line sets one setting of at a time: --set data.<call>.<setting>=... for a call’s (data.token_windows.seq_len=512) and --set data.source.<field>=... for its store’s.

The resolved config is the run’s spec. A recipe writes it to run.json next to the checkpoints with save, and load reads it back into the same class, so inference rebuilds a run from what training was built from. A field the file lacks takes its declared default, and a field the class does not have raises an error. The model and the objective record their classes’ defaults too (ModelConfig.defaults), so a default changed since leaves the record building what it built.

NameSummary
DecoderRunConfigA run over a loaded decoder, which may run a block of its layers in passes: the pass count, the block, the passes trained through and their recomputation (CausalTransformer.with_passes), each the run’s choice over the checkpoint’s own.
MetricConfigA metric’s class and the arguments the run gives it.
ObjectiveConfigAn objective’s class and the arguments the run gives it.
PreparedWhat a run trains, as its class builds it (RunConfig.prepare).
RunConfigA whole run’s configuration.
SourceOptionsWhat a run builds of the source it starts from: Pretrained.load’s keywords for its parts, beside where it reads them (TrainerConfig.source_placement), every run kind’s alike (RunConfig.source_keywords).
TrainerConfigHolds the run length, checkpointing, sharding and run tracking.
flag_selectedgiven command-line arguments with each optional setting of selected, a field’s name and its subcommand’s, turned on by its own flags: --lora.rank 16 stands for lora:lora --lora.rank 16, where no lora: choice is given.
loopingA loaded checkpoint whose block runs steps passes: a looped one’s own block, or layers, which gives a decoder with no loop one over its own layers.

dataclass source

class DecoderRunConfig(
model: ModelConfig = ModelConfig(),
data: Annotated[Dataset | None, tyro.conf.Suppress] = None,
validation: Annotated[Dataset | Mapping[str, Dataset] | None, tyro.conf.Suppress] = None,
data_seed: int = 0,
data_plan_version: int | None = None,
source: SourceOptions = SourceOptions(),
optim: OptimConfig = OptimConfig(),
trainer: TrainerConfig = TrainerConfig(),
objective: ObjectiveConfig | None = None,
lora: Annotated[LoRA, tyro.conf.subcommand(lora)] | None = None,
val_metrics: Annotated[tuple[MetricConfig, ...], record_argument(tuple[MetricConfig, ...])] | None = None,
val_every: Annotated[Mapping[str, int | str], record_argument(Mapping[str, int | str])] = dict(),
val_summaries: Annotated[tuple[Summary, ...], record_argument(tuple[Summary, ...])] = (),
loop_steps: int | None = None,
loop_layers: tuple[int, int] | None = None,
loop_backprop_steps: int | None = None,
loop_remat: bool = False,
)

A run over a loaded decoder, which may run a block of its layers in passes: the pass count, the block, the passes trained through and their recomputation (CausalTransformer.with_passes), each the run’s choice over the checkpoint’s own. The language-model and the decision runs take it.

loop_steps: int | None

The passes a looped checkpoint runs its block (CausalTransformer.loop), such as Ouro’s at 1, 2 or 4; None keeps the checkpoint’s own.

loop_layers: tuple[int, int] | None

The block of layers [first, end) that loop_steps repeats: a pretrained decoder with no loop gains one over its own layers, as Qwen 3.5’s middle block repeated twice; None keeps a looped checkpoint’s block.

loop_backprop_steps: int | None

Train through the last k passes of the loop alone, with earlier passes cut from backpropagation; None keeps the loop’s own choice.

loop_remat: bool

Checkpoint every pass of the loop, including the first, and recompute it in the backward pass; requires a loop and leaves decoding unchanged. False keeps a checkpoint’s own choice.

loops: bool

Whether the run asks anything of a loop (looped).

def looped(start: Pretrained) -> Pretrained

start with its block run as the run asks (looping), as loaded where it asks nothing.

dataclass source

class MetricConfig(
name: str,
fields: JsonDict = dict(),
defaults: JsonDict = dict(),
split: str | None = None,
)

A metric’s class and the arguments the run gives it.

name is an alias of dew.registry.metrics or an import path, held as the path. fields are the constructor arguments the run states, as ObjectiveConfig holds an objective’s: a CLIP metric’s modelname, an FID’s weights. defaults holds every other argument the class defaults, so a metric whose default model changes later scores a recorded run as it did. split is the validation split it scores, every split where None.

def build(**derived={}) -> Metric

Build the metric from its arguments and those the run derived where it states none of its own (the field a video run’s sample is read from, say).

dataclass source

class ObjectiveConfig(name: str, fields: JsonDict = dict(), defaults: JsonDict = dict())

An objective’s class and the arguments the run gives it.

name is an alias or an import path, held as the path. fields are the constructor arguments the run states, held as their records (registry.argument_records), which the command line sets one flag each (--objective.ema_decay 0.999); what the objective is built around, the model and what a recipe reads off its data, build takes from the caller. defaults holds every other argument the class defaults, as ModelConfig holds a model’s, so an EMA decay, a loss weighting or a sampling step count the class defaults differently later leaves a recorded run as it was.

def build(**derived={}) -> Objective

Build the objective from its arguments and those the caller derived, which replace a recorded default; an argument the run states is refused there, since the caller’s would silently replace it.

dataclass source

class Prepared(
run: RunConfig,
objective: Objective[LossT, EffectsT],
dataset: Dataset | Loader,
metrics: Sequence[Metric] = (),
rollout: Rollout | None = None,
summary: Mapping[str, object] | None = None,
validation: Dataset | Mapping[str, Dataset | Split] | None = None,
after: Callable[[TrainState, str], None] | None = None,
)

What a run trains, as its class builds it (RunConfig.prepare).

run is the run as it records itself, with what building resolved (a pinned source, the model’s fields as built); objective and dataset what it trains on, and the rest what RunConfig.train takes beside them. train trains it under a name, and inspected views one sampled update of it there (RunConfig.viewed).

def train(name: str) -> TrainState
def inspected(name: str) -> dict[str, float]

dataclass source

class RunConfig(
model: ModelConfig = ModelConfig(),
data: Annotated[Dataset | None, tyro.conf.Suppress] = None,
validation: Annotated[Dataset | Mapping[str, Dataset] | None, tyro.conf.Suppress] = None,
data_seed: int = 0,
data_plan_version: int | None = None,
source: SourceOptions = SourceOptions(),
optim: OptimConfig = OptimConfig(),
trainer: TrainerConfig = TrainerConfig(),
objective: ObjectiveConfig | None = None,
lora: Annotated[LoRA, tyro.conf.subcommand(lora)] | None = None,
val_metrics: Annotated[tuple[MetricConfig, ...], record_argument(tuple[MetricConfig, ...])] | None = None,
val_every: Annotated[Mapping[str, int | str], record_argument(Mapping[str, int | str])] = dict(),
val_summaries: Annotated[tuple[Summary, ...], record_argument(tuple[Summary, ...])] = (),
)

A whole run’s configuration. A kind of run subclasses it with its own settings and builds what it trains from them (build_run, which prepare calls on the run with its Hub sources pinned); run trains it.

RESUMABLE: tuple[str, ...]

The fields a run resumed under its name may change: how it trains and optimizes, what it reads and scores, and, in a kind of run that adds them, how it previews. Every other field says what it trains, which the resume must keep (check_resume), and so does trainer.quantization, which changes the forward pass the weights train through.

BUILT: tuple[str, ...]

The fields, by their dotted paths, a run may leave None for its build or its pinning to resolve, whose resolution a later Dew, or a later state of what it reads, may change: the storage of the source it reads (SourceOptions.param_dtype), the step of a saved Dew run it starts from, which that run’s later checkpoints move (with_pinned_run), the data plan version its order is read by (data_plan_version), and in a kind of run that adds them its own (a decision run’s option text). A run that leaves one None and resumes a saved run under its name takes the saved run’s value (resumed), so running the same command again resumes what it trained; one it sets is its own, and a resume refuses it where it differs (check_resume). A saved run whose record holds one None left it to its build unresolved, which nothing it saved resolves (each earlier shape’s translation resolves what it can, earlier_run), so the value this build resolves is its value.

UNSEEDED: str

How a run of this kind reads a mix saved unseeded and seeded now, as the refusal of its resume names it (PlanOrderChanged.routed): the data layer’s mix(..., seed=None); a decision run’s mixture’s order (dew.decision.data.SCHEDULE_MIXTURE).

data: Annotated[Dataset | None, tyro.conf.Suppress]

The data the run trains on: a chain ending in its preset (dew.data.presets), whose batch stage is the run’s batch, read and loaded as it says. The record writes it as its chain (dew.data.records); on the command line, --set data.<preset>.<setting>= sets one setting of a recorded chain’s call.

validation: Annotated[Dataset | Mapping[str, Dataset] | None, tyro.conf.Suppress]

The splits each evaluation scores, one dataset or several by name, each batched by its own batch stage or the training batch.

data_seed: int

The seed of the data’s order and of its random steps.

data_plan_version: int | None

The data plan version the run reads its order by (dew.data.plan.PLAN_VERSIONS): None for the saved run’s it resumes, or else this Dew’s, which resumed resolves on every path a run trains by and its record holds, so a resume reads the order it read whatever a later Dew’s is (BUILT); a saved run that recorded none read by version 8, and one that recorded a data_plan of None by 9 (earlier_data_plan).

source: SourceOptions

What the run builds of the source it starts from (--source.text-only, --source.prediction-depths), where it starts from one (pretrained).

objective: ObjectiveConfig | None

The objective’s class and the arguments the run states, which train names from the objective it trains when the run leaves it unset.

lora: Annotated[LoRA, tyro.conf.subcommand(lora)] | None

The low-rank adapter the run trains instead of the whole model.

On the command line, set it with --lora.rank 16 --lora.modules q_proj v_proj (see cli). A recipe attaches it to the source that --pretrained loads (Pretrained.adapt), or, when training from scratch, to a model freshly initialized from the run’s key. The objective then trains only the adapter’s factors, and the run’s record stores the model with the adapter attached.

val_metrics: Annotated[tuple[MetricConfig, ...], record_argument(tuple[MetricConfig, ...])] | None

The metrics each validation split scores, each with the arguments the run gives it (a CLIP metric’s model, an FID’s weights) and the split it scores, every split where it names none (MetricConfig.split), one JSON list on the command line: --val-metrics '[{"name": "accuracy"}, {"name": "brier", "split": "test"}]' scores accuracy on every split and the Brier score on test. A split no metric names scores none; [] scores none anywhere. None scores the kind of run’s own on every split (default_metrics): perplexity for a language model, a decision run’s accuracy, calibration, risk-coverage and log loss, a JEPA run’s probes, CLIP for an image run. A metric given is asked for whatever the data holds: a split the run does not validate, or data with no validation split, refuses it.

val_every: Annotated[Mapping[str, int | str], record_argument(Mapping[str, int | str])]

A split’s own validation cadence, as trainer.eval_every spells one (EvalSuite.every), one JSON object on the command line: --val-every '{"test": "25%"}' scores test every quarter of the run and every other split at trainer.eval_every. A split it names must be one the run validates.

val_summaries: Annotated[tuple[Summary, ...], record_argument(tuple[Summary, ...])]

Scalars weighed from each evaluation’s split scores, a board’s weighted index, each formed at a step that scored every split it weighs and logged as evaluation/summary/<name>, which --trainer.best evaluation/summary/<name> ranks by (dew.training.evaluation.Summary): --val-summaries '[{"name": "board", "weights": {"val/accuracy": 1, "test/accuracy": 2}, "better": "higher"}]'.

def to_dict() -> dict[str, JSON]

Return a JSON-safe record of the run.

A value of a subclass of the field’s type is written as its class’s import path and its fields. The data and its validation splits are their chains’ records (dew.data.records.encode), and a chain the codec cannot describe is {"unrecorded": reason}.

def unrecorded() -> list[str]

Why each of the run’s chains is not recorded, none where its record can rebuild them all: a run built in Python with a step the record cannot describe trains as built, and its run.json, which says why, refuses to rebuild it.

def from_dict(values: Mapping[str, object]) -> Self

Read back what to_dict wrote, for subclasses too. A field the record lacks takes its default; an unknown field raises. A chain recorded as unrecorded, or a recipe’s record written before runs recorded chains, is refused by name. A model recorded with an earlier main’s fields reads as its checkpoint’s record does.

def override(point: Mapping[str, JSON]) -> Self

This config with each dotted path in point replaced, through its record.

A path walks the run’s fields and the fields of the values they hold. Under model or objective, a name the config does not declare is an argument of the class it names, as --model.num_layers is on the command line. A path the run does not declare raises, rather than training the unchanged config. The paths are set together and the run is built once, so fields valid only together (a token budget with no row batch) are one point; each path is placed in this config, so a path under model names an argument of the model this config holds.

def assigned(assignments: Sequence[str]) -> Self

This config with each path=value in assignments set (override).

The value is read as the field’s annotation reads a flag: a number, a string, a literal or a tuple of them as its type parses it on the command line, None for an optional field, and anything else (a record, a list of records) as JSON, or as the text itself when it is not JSON.

def save(directory: str) -> str

Write this config as run.json in directory and return the path.

The file is the run’s class record, its class’s import path and its fields (to_dict), so load rebuilds the run as the class it is. The path goes through epath, the same filesystem layer Orbax writes the checkpoints with, so a gs:// run directory gets the record too.

def record() -> dict[str, JSON]

The run’s class record: its class by its alias in dew.registry.runs, or by its import path where the table holds none, and its fields (to_dict).

def source_keywords(*, frozen: bool) -> SourceOptionsKeywords

Pretrained.load’s keywords for the source the run starts from: what it builds of it (source), where it reads it (TrainerConfig.source_placement, frozen where the run keeps the source as it is) and what it stores its parameters in. Every run kind that starts from a source loads it with these.

A source the run trains is stored in its numerics’ dtypes.params, the master weights its updates land in. One it keeps frozen is stored as its checkpoint stores it (param_dtype='auto'), since no update reaches it: a float32 copy of a bf16 checkpoint doubles its memory and is cast back to bf16 every pass for nothing, while a cast to the compute dtype would round the float32 tensors a bf16 checkpoint keeps (Mamba’s A_log). A dtypes.params narrower than float32 stores it so too. A storage the run’s source options state wins (SourceOptions.param_dtype).

def source_attention() -> AttentionImpl

The attention implementation a run that starts from a source loads it with: its model’s attention_impl field, auto unless set, refused unless it names one (AttentionImpl). Every run kind that loads a source reads it here.

def built_fresh() -> None

Refuse source in a run whose model is built from model, loading no source: what it says Pretrained.load builds of a source (source_keywords) reaches nothing such a run builds.

def sourced_model() -> None

Refuse a model name in a run whose source decides its architecture: such a run reads its model’s fields’ few choices alone, and a name other than its kind’s own would be dropped.

def check_resume(saved: Mapping[str, object]) -> None

Refuse to resume the run whose record is saved as this one where they train different things: another class, or a field outside RESUMABLE that differs. A changed rope_theta keeps every shape, so it would otherwise continue the old weights under new semantics. The saved fields are read back as this class reads them (from_dict), so a field the record predates takes its default, and one it left to its build unresolved takes this build’s.

def cli(args: Sequence[str] | None = None, *, default: Self | None = None) -> Self

Parse a run from args, or from the process’s own command line when args is None.

An optional setting such as lora is turned on by its own flags. For example, --lora.rank 16 --lora.modules q_proj stands for lora:lora --lora.rank 16 ..., and with no --lora. flag the run trains without an adapter.

--model <alias or import path> picks the model’s class, and every field the class declares is a flag, --model.<field>, typed by its annotation (ModelConfig). Naming the class the run defaults to keeps the default’s fields; naming another starts from that class’s own and the default’s compute dtype. --objective picks the objective’s class the same way, each argument it takes a flag, --objective.<argument> (ObjectiveConfig). --set path=value, once a setting, sets a field at its dotted path after the flags, as dew train does (assigned): the data’s settings are its chain’s calls’ (--set data.lm.seq_len=512), which have no flags of their own.

def load(directory: str, *, trust: Sequence[str] = ()) -> Self

Read the run a directory’s run.json records, or the file itself, as the class it names, which must be this one or derive from it. trust names the packages outside Dew the record may import.

def read(record: Mapping[str, object], *, trust: Sequence[str] = ()) -> Self

The run a class record (record) names, as its class, which must be this one or derive from it.

def prepare() -> Prepared

Build what this run trains, as build_run builds it from the run with the fields it leaves to its build at the run it resumes (resumed) and its Hub sources pinned (pinned), so the record names what it started from and is what it loads.

def resumed(paths: Sequence[str] | None = None) -> Self

This run with each field it leaves None for its build to resolve (BUILT, or those of them paths names) at the value the saved run it resumes under its name recorded, read as every earlier shape of a saved run is (earlier_run), and its data plan version, where it resumes no run that recorded one, this Dew’s (PLAN_VERSIONS): the one place a run’s version is resolved, on each path a run trains by (prepare, and readied, which a run trained in Python reaches alone), before its data compiles or its record is written. Its other fields as they are where it resumes no run (TrainerConfig.resumes) or leaves none to its build.

def build_run() -> Prepared

Build what this run trains, the run as it stands, its sources already pinned (prepare pins them first): the objective its record names around its model, on its data. A kind of run with settings of its own builds from them in its own build_run.

def default_metrics(*, validates: bool) -> tuple[MetricConfig, ...]

The metrics a validation pass scores where val_metrics names none, by what the run’s data validates: a kind of run’s own, none for a run that names its objective alone.

def metric(config: MetricConfig) -> Metric

config built with what this kind of run derives for it where it states none of its own.

def validation_metrics(*, validates: bool) -> tuple[Metric, ...]

The kind of run’s own metrics (default_metrics), built (metric), by what the run’s data validates: what each split scores where val_metrics names none (suites).

def suites(
dataset: Loader,
validation: Mapping[str, Split] | None,
metrics: Sequence[Metric] = (),
) -> Mapping[str, EvalSuite] | None

Each split the run validates, validation’s or dataset.val as val, with the metrics val_metrics names for it built (metric), else the kind’s own metrics, at its cadence in val_every, else the run’s; None where neither names any, so each scores the kind’s own at the run’s. A metric or a cadence for a split the run does not validate is refused.

def chain() -> Dataset

The run’s data. Raises ValueError for a run that names none: a run trains on the chain it states (dew.data.presets), in Python or in its run.json.

def compiled(
objective: ObjectiveData,
dataset: Dataset | Loader,
validation: Dataset | Mapping[str, Dataset | Entry] | None,
) -> tuple[Loader, Mapping[str, Reader | Entry] | None]

dataset and validation as the run trains on them (fitted): compiled with objective’s stages, data_seed and data_plan_version, the training read bounded to trainer.examples where the run counts them, or to trainer.epochs passes over its source examples where its read has no record count, so its length is read off the loader this returns.

def run() -> TrainState

Prepare the process, build what this run trains (prepare) and train it.

The run trains under trainer.name, or <objective>-<data>/date-<time>.

def pinned() -> Self

This run with its Hub sources named at the commits they resolve to now, as its record names them and prepare loads them: here an imported adapter’s (LoRA.resolved); a kind of run pins its own sources beside it. Itself where nothing is left to pin.

def with_pinned_run(pretrained: str | None) -> Self

This run with the checkpoint it reads of the saved Dew run it starts from (pretrained, a run’s directory) pinned to the step its selection resolves to now (SourceOptions.step: best, best:<metric> or its task’s default), so its record names the run and the step it read and a resume reads them again. Itself where it starts from no saved run, or names a step.

def profile(
directory: str | None = None,
*,
warmup: int = 20,
steps: int = 10,
capture: CaptureOptions | None = None,
) -> ProfileReport | None

Train this run afresh under directory for warmup steps, then a window of steps more traced (ProfileWindow), and report where the window’s steps spent their time (ProfileReport), written as report.json beside XProf’s own reports. The warmup is whole windows, at least one, so the step compiles before the window and the window is one log interval. The run keeps its own settings but those that would read or write outside the window, which a profile drops together: its length, stop, checkpoints local or persistent, the checkpoint ranking (best) and the best model it would end on, its cooldowns, the validation splits it names and their metrics, cadences and summaries (a split its chain holds out scores after the window), its W&B sink, and a lowering, a compile ahead or a step compiled ahead for its own length, which the profile trains past. directory unset is profiles/<the run’s name>/<the time it starts>, read once the process is prepared, as run reads a run’s name. The run’s own directory is left as it is. capture is what the capture records (CaptureOptions), its own by default. Raises ValueError for a directory that holds this run’s profile already. The report is process zero’s, which logs the run; None on the others.

def recorded(objective: Objective[Loss, Effects]) -> Self

Return this config as the record of the run that trains objective.

The record names the class by its import path, so it says what was trained and loads in a process that imported nothing. Arguments the run states belong to the class it names, so a run that states them for another class than the one it trains is refused. Each argument the objective resolved from what it was built around (Objective.resolved) and the run leaves unstated is recorded as the objective holds it, so the record builds what the run sampled with.

def check_process() -> None

Refuse to train in a process that does not carry the run’s process options (dew.training.runtime.unprepared): run() and dew train set them up before JAX opens a backend (prepare_process), and a run trained after prepare() in a process nothing set up with them would drop them.

def readied(
objective: Objective[Loss, Effects],
dataset: Dataset | Loader,
validation: Dataset | Mapping[str, Dataset | Split] | None,
metrics: Sequence[Metric],
name: str,
) -> tuple[Self, Loader, Mapping[str, Split] | None, Mapping[str, EvalSuite] | None, int | None]

This run as it trains objective under name, and what it trains on, before anything is placed (train, viewed): its data compiled with the objective’s own stages and its validation splits with it, the suites they score (suites), sized and ramped (TrainerConfig.resolved), with its steps; its trunk quantized where it asks; its record naming the run’s directory and its optimizer’s groups (grouped).

def train(
objective: Objective[Loss, Effects],
dataset: Dataset | Loader,
*,
name: str,
metrics: Sequence[Metric] = (),
rollout: Rollout | None = None,
summary: Mapping[str, object] | None = None,
validation: Dataset | Mapping[str, Dataset | Split] | None = None,
after: Callable[[TrainState, str], None] | None = None,
) -> TrainState

Train objective on dataset as this config describes; every kind of run calls this after building both.

The run lives under name in trainer.checkpoint_dir, and process zero writes the record there before training starts. A trainer.wandb opens a tracker under the same name, with the record, summary (the recipe’s own view of the run) and the step count as its config. Only trainer.publish publishes the final checkpoint to the W&B registry, under the name with slashes and spaces replaced because an artifact name allows neither. Local tracking journals live in a separate tracking directory.

dataset is compiled once with the objective’s own stages (dew.data.compile.fitted), and its validation splits with it; the run’s length, its ramp, its cadences and its recorded plan are read off that one compiled loader, which fit trains on. Its batch stage is the run’s batch, and a token budget’s accumulation is trainer.update_tokens over its token batch.

A trainer.quantization wraps the module objective trains before anything initializes it, so the run learns through the quantized forward pass. A lora is not applied here: the recipe attaches it to the model and variables it builds the objective from. rollout is the trainer’s rollout, which turns each prefetched batch into the one the step trains on, as an on-policy objective samples it. validation names the splits each evaluation scores in place of dataset.val, as Trainer.fit takes them: a held-out split and a test set are logged as val/... and test/.... metrics are the kind’s own, which each split scores where val_metrics names none. With no trainer.best, the first split’s loss ranks the checkpoints; a Best over a metric names its split. after runs on the state training ends on and the run’s directory before the checkpoint is published, as a decision run fits its calibration there, so what it writes is published.

def compiled_ahead(
objective: Objective[LossT, EffectsT],
dataset: Loader,
steps: int | None,
rollout: Rollout | None,
name: str,
) -> CompiledAhead

The run’s first step compiled for its compile_ahead target and saved (AheadTarget.compiled), its trainer built as train builds it, its links the run’s own, over that target’s compile-only devices (AheadTarget.compile_only), with no tracker: what ends a run that compiles ahead. A run name whose directory holds its checkpoints reads them, and compiles the step its resume takes; one that has none writes none.

def viewed(
objective: Objective[Loss, Effects],
dataset: Dataset | Loader,
*,
name: str,
rollout: Rollout | None = None,
validation: Dataset | Mapping[str, Dataset | Split] | None = None,
) -> dict[str, float]

The run’s inspection of one sampled update over the batch its first step takes (dew.training.inspected, first_step_batch), the run readied as train readies it (readied) and its trainer built as train builds it, from the checkpoints its directory name holds, their latest, else its start, with no tracker: what inspected returns.

def inspected() -> dict[str, float]

What the run’s inspection (TrainerConfig.inspection) logs of one sampled update over its training data’s first batch, from its checkpoints’ latest, else its start, offline (viewed): the view its training logs, for a saved run, which this trains and writes nothing of (dew inspect).

def grouped(objective: Objective[Loss, Effects], name: str) -> OptimConfig

The run’s optimizer as it trains under name: each group it names that claims the modules its objective draws fresh beside a loaded base matching those that hold a parameter (ParamGroup.fresh, trained_modules). Where it names none, a resumed run trains in the groups its record names, as its optimizer state holds them (none for a record written before runs took them), and a new run that draws such modules in their default groups (OptimConfig.fresh_groups). Every process agrees on the groups before any records them.

def built(
objective: Objective[LossT, EffectsT],
mesh: MeshSpec | jax.sharding.Mesh,
rollout: Rollout | None,
**resources={},
) -> Trainer[LossT, EffectsT]

The run’s trainer over mesh: how it steps, every numeric choice its TrainerConfig holds (key, microbatches, layout, accumulation, split, loss scale, kernel tuning), and the resources its mode adds (checkpoints, tracker, profile, a lowering or a step compiled ahead), so a run that trains and one that compiles ahead build the one trainer. The trainer is the training package’s lazy export (dew.training), read where a run trains or compiles ahead, so a config read alone reads none of it.

def compute_dtype() -> str

The dtype the run’s model computes in: the one its record states, else its numerics’ (TrainerConfig.precision).

def drawn_dtype() -> str | None

The dtype the run stores the parameters it draws fresh in (Trainer’s param_dtype): its numerics’ dtypes.params where that is not Flax’s own float32 and its model’s record states none; None leaves each module’s own.

def numerics_fields() -> dict[str, str]

The fields the run’s numerics derive for its model (ModelConfig.build): its compute dtype, where the model takes one and its record states none.

def sweep(
space: Space,
*,
train: Callable[[Self], float],
trials: int,
ledger: str | Path,
tracker: Tracker,
search: Search = _RANDOM_SEARCH,
) -> list[TrialFinished]

Train trials trials of this config over space and return the ledger.

Each trial draws a point from space and trains under the run name <trainer.name>/trial-<index>, so trials keep their own checkpoints and tracking, and the score train returns is recorded for it. tracker receives that score as sweep/value at the trial’s number, along with the trial’s TrialFinished record. A trial is written to ledger before it is reported, so rerunning the same call continues an interrupted sweep.

dataclass source

class SourceOptions(
text_only: bool = False,
prediction_depths: int | None = None,
attention: dict[str, Literal['causal', 'block_bidirectional']] | None = None,
dex: Annotated[DifferentialExtension, record_argument(DifferentialExtension)] | None = None,
scan_layers: bool = False,
param_dtype: str | None = None,
step: int | str | None = None,
ema: bool | None = None,
)

What a run builds of the source it starts from: Pretrained.load’s keywords for its parts, beside where it reads them (TrainerConfig.source_placement), every run kind’s alike (RunConfig.source_keywords).

text_only: bool

A multimodal wrapper’s language model alone: no tower, projector or audio encoder built, nor their tensors read (Pretrained.load’s text_only).

prediction_depths: int | None

The first that many multi-token prediction depths, 0 for none; None the checkpoint’s own (Pretrained.load’s prediction_depths).

attention: dict[str, Literal['causal', 'block_bidirectional']] | None

How each layer kind’s attention reads, by kind (full_attention): causal, or block_bidirectional, each of a row’s blocks read whole, which a run’s rows then lie in (Pretrained.load’s modeling_overrides); None the checkpoint’s own: --source.attention full_attention block_bidirectional.

dex: Annotated[DifferentialExtension, record_argument(DifferentialExtension)] | None

DEX’s differential on chosen heads (dew.nn.dex, Pretrained.load’s dex), its record as JSON: --source.dex '{"layers": [[0, [1, 3]], [5, [0, 2]]], "anneal_steps": 1000}', its trains the paper’s recipe unless “all”; None for none.

scan_layers: bool

The decoder’s runs of like layers, and of layers repeating one period, under flax’s scan (Pretrained.load’s scan_layers): a decode step’s program grows with the period, not the depth.

param_dtype: str | None

The dtype the source’s parameters are stored in, its checkpoint’s own as ‘auto’; None stores a source the run trains in its numerics’ dtypes.params and one it keeps frozen as its checkpoint does (RunConfig.source_keywords). A frozen bfloat16 base beside a float32 head and adapter is --source.param-dtype bfloat16.

step: int | str | None

Where the run starts from a saved Dew run (--pretrained <run directory>), the checkpoint it reads: a step, latest, best, best:<metric>, or None for the one the run’s task loads by default (dew.interop.saved_sources.released_run). The run records the step it read (RunConfig.pinned).

ema: bool | None

Where the run starts from a saved Dew run, whether it reads that run’s EMA weights; None its task’s own choice (SavedRun.variables).

def overrides() -> Mapping[str, object] | None

The modeling_overrides these options state, None where they state none.

def keywords() -> SourceBuild

Pretrained.load’s keywords for what a run builds of its source. With weights=False beside them a load is the run’s base as the run builds it, its config and its variables’ shapes, read from the source’s config and safetensors headers alone, no weight downloaded (a release of PyTorch pickles alone, whose shapes no header states, is read whole): a check that a base loads under a run’s options, on a disk too small for its weights.

dataclass source

class TrainerConfig(
name: str | None = None,
checkpoint_dir: str = './checkpoints',
keep: Annotated[int | Keep, _keep_argument()] = Keep.latest,
resume: Resume = 'auto',
best: Annotated[str | Best | tuple[Best, ...] | None, _best_argument()] = None,
update_tokens: int | None = None,
key: int = 0,
steps: int | None = None,
epochs: int | None = None,
tokens: int | None = None,
examples: int | None = None,
stop: Annotated[Pause, tyro.conf.subcommand(pause)] | Annotated[Plateau, tyro.conf.subcommand(plateau)] | None = None,
restore_best: bool = False,
cooldowns: Annotated[Cooldowns, tyro.conf.subcommand(cooldowns)] | None = None,
log_every: Annotated[int | str, _cadence_argument(timed=False)] = 100,
eval_every: Annotated[int | str | None, _cadence_argument(timed=False)] = 'epoch',
checkpoint_every: Annotated[int | str | datetime.timedelta | None, _cadence_argument()] = 'epoch',
checkpoint_final: bool = True,
local_directory: str | None = None,
local_every: int | str | None = None,
scored_to: str | None = None,
accumulation: int = 1,
split: int | None = None,
batch_ramp: Ramp | None = None,
dynamic_scale: bool | None = None,
precision: Annotated[Numerics, _numerics_argument()] = Numerics(),
mesh: MeshSpec = MeshSpec(),
microbatches: int | None = None,
layout: Layout = Layout(),
profile: ProfileWindow | None = None,
run_profile: RunProfile | None = RunProfile(),
compilation_cache_dir: str | None = None,
wandb: Wandb | None = None,
publish: bool = False,
multi_host: bool | None = None,
xla_flags: str | None = None,
quantization: Quantization | None = None,
lower_for: str | None = None,
compile_ahead: Annotated[AheadTarget, tyro.conf.subcommand(ahead)] | None = None,
ahead: str | None = None,
inspection: Annotated[Inspection, tyro.conf.subcommand(inspect)] | None = None,
diloco: Annotated[DiLoCo, tyro.conf.subcommand(diloco)] | None = None,
links: Annotated[Mapping[str, Link], record_argument(Mapping[str, Link])] = dict(),
tune_kernels: bool = False,
)

Holds the run length, checkpointing, sharding and run tracking.

keep: Annotated[int | Keep, _keep_argument()]

The number of latest checkpoints kept, besides the best one (Keep’s own default).

resume: Resume

Whether the run continues its directory’s latest checkpoint: where there is one, “require” for a job that must not start afresh, or “never” for one that must not continue another (Trainer.fit).

best: Annotated[str | Best | tuple[Best, ...] | None, _best_argument()]

The metric name and ranking policy; None selects validation loss or training loss.

update_tokens: int | None

The optimizer update’s token budget, a whole number of the data’s token batches (its batch stage’s TokenBudget), which sets the run’s accumulation (RunConfig.train).

key: int

The seed of the run key, which sets parameter init and every per-step draw.

epochs: int | None

The run length as passes over the data; steps sets it directly instead.

tokens: int | None

Stop after the loss has scored this many targets across all microbatches, including before a resume.

Counts the tokens committed updates trained on, reported as train/committed_tokens, without padding. The final microbatch may cross the budget. With steps, epochs or examples, the first cap reached ends the run. A token-only run needs a constant learning rate or a schedule with its own explicit step horizon.

examples: int | None

The run length as the first this many source examples of the data, each read once with every row it makes: the records the chain makes in its own order, counted before they are packed and before an objective makes several rows of one (a decision layout’s row a question) (Trainer.fit’s examples); steps or epochs set it instead.

stop: Annotated[Pause, tyro.conf.subcommand(pause)] | Annotated[Plateau, tyro.conf.subcommand(plateau)] | None

End the run at a fraction of its length, to resume it later as if it had never stopped (Pause): --trainer.stop.fraction 0.5; or once a metric, by its recorded name as best names one, stops improving (Plateau): trainer.stop:plateau --trainer.stop.metric val/loss --trainer.stop.evals 3.

restore_best: bool

End on the model the run ranked best (Trainer.fit’s restore_best): the state the run hands on (RunConfig.train’s after) holds the best step’s weights, EMA and counters, with the final step’s optimizer state. The checkpoints stay as written; dew export --step best exports the best one.

cooldowns: Annotated[Cooldowns, tyro.conf.subcommand(cooldowns)] | None

Branch the run into cooldowns, each scored by its validation and discarded (Cooldowns): --trainer.cooldowns.at 20% 40% --trainer.cooldowns.length 20%.

log_every: Annotated[int | str, _cadence_argument(timed=False)]

The steps between logs, or a share of the run such as “1%” (eval_every).

eval_every: Annotated[int | str | None, _cadence_argument(timed=False)]

The steps between validation passes.

It is a number of steps, “epoch” for one pass over the data, a share of the run such as “5%”, resolved once against its planned length in steps, epochs, examples or tokens (dew.training.Trainer.fit), or None to never validate. “epoch” over a stream that reports no record count raises a ValueError, since such a stream has no pass.

checkpoint_every: Annotated[int | str | datetime.timedelta | None, _cadence_argument()]

The steps between checkpoints, with the same choices as eval_every, or a duration such as 30m.

A stream whose iterator cannot report its read position must train with None; the trainer refuses any other value for it.

checkpoint_final: bool

Write the final state when training ends, independently of checkpoint_every.

False disables this save; cadenced, ranked and preemption saves keep their own policy.

local_directory: str | None

A path on every host’s own disk where the run also writes checkpoints every local_every steps, under its name as in checkpoint_dir, cheaper than the persistent ones and lost with the host (Checkpoints); a resume reads the newer of the two. Set with local_every, or neither.

local_every: int | str | None

The steps between local checkpoints, or a share of the run such as “5%” (local_directory).

scored_to: str | None

A directory that takes every validation batch’s scoring artifacts at each evaluation, a decision run’s decisions among them, as <scored_to>/step-<step>/<split>-<batch>.npz, for pooling a pass’s metrics offline (Trainer.fit’s scored_to); None writes none.

accumulation: int

The number of steps per optimizer update.

split: int | None

The microbatches each step’s batch is read in, its rows split evenly.

None lets the trainer plan it from the compiled step’s memory: the fewest that fit the devices before it recomputes anything, at the least recomputation that fits (Trainer.compile). A step’s tokens and the update’s stay as configured either way; only the microbatch shrinks.

batch_ramp: Ramp | None

A ramp that grows batch_size over the run’s first records instead of starting there.

It sets the global batch the run starts at, what each stage adds, and the records the whole ramp spans. Unset trains at batch_size throughout. Either way there is one optimizer update a step, and the compiled step is traced once per stage.

dynamic_scale: bool | None

Whether the run scales its loss dynamically (flax.training.dynamic_scale.DynamicScale): None for a float16 compute’s alone, which needs it (RunConfig.built); a run states True or False to choose, as a run saved before its numerics records the one it trained under.

precision: Annotated[Numerics, _numerics_argument()]

The run’s numerics, set once and recorded with it (dew.nn.precision.Numerics; ACCURATE for scientific runs): its dtypes by Lightning’s names, its products’ precision by JAX’s levels and its ops’ forms, over every program it traces, the step and its evaluations (dew.nn.kernels.generation.run_precision). Its compute dtype is the model’s where the model’s record states none (RunConfig.numerics_fields), its parameters’ a loaded checkpoint’s storage, and a float16 compute scales the loss (RunConfig.built). The optimizer’s moments are a storage format of their own (OptimConfig.state_dtype: float32 under any weights, bfloat16 stochastically rounded, int8 blockwise). A dtype or precision the model states still wins, and a resumed run traces under the numerics it records.

microbatches: int | None

Microbatches a pipelined step feeds through the mesh’s stages, a multiple of mesh.stage; unset feeds one a stage.

profile: ProfileWindow | None

One profiler window: the steps to trace, the warmup before them and the output directory, in place of the run’s own (run_profile). Unset traces the run’s own window alone.

run_profile: RunProfile | None

The run’s own profile window, every run’s by default: two of its steps traced once ten have run past its first compile, kept with its records (window), its trace and manifest alone, where the process can take a capture (dew.training.profiling.capturable): a capture’s fixed cost, about a second on a CPU host whatever its steps (row 4112), a run of minutes does not see, and a run shorter than its warmup takes none. A profile the run names replaces it; None traces none of its own.

compilation_cache_dir: str | None

The directory of JAX’s persistent compilation cache, so a restart skips recompiling the step.

None caches under the user’s cache directory wherever the run’s processes share it (dew.training.runtime.ready_process); a pool across hosts without shared storage names one they share. JAX_ENABLE_COMPILATION_CACHE=false compiles from scratch every run.

wandb: Wandb | None

An optional W&B telemetry sink, in addition to the local tracking journal.

Scalars, records and objective previews are tracked; checkpoints are not uploaded.

publish: bool

Publish the final checkpoint to W&B; requires wandb, online, and checkpoint_final.

multi_host: bool | None

Whether to join the JAX process pool.

None tries, and continues alone only when no cluster is configured; True requires the pool; False never joins.

xla_flags: str | None

Extra XLA_FLAGS for this run, appended to the environment by prepare_process.

dew train sets them in XLA_FLAGS before the process loads jaxlib, so they hold for every runtime XLA loads (dew._devices.hold_xla_flags); prepare_process appends them before JAX opens a backend, unproven where jaxlib has loaded. Library users set XLA_FLAGS before the process starts to make them certain; docs/performance.md records what was measured.

quantization: Quantization | None

The quantized-training spec, wrapped around the module the objective trains.

The wrapper is applied before the run initializes the module. Unset trains in the compute dtype. dew.quantization describes what the wrapper does and what it keeps.

lower_for: str | None

Check the run’s step for a GPU on any host, a CPU too, then exit without training.

--trainer.lower-for sm80 (or sm75, sm86, sm89, sm90, sm120) builds the run as it would, traces its first step as on a GPU of that CUDA generation, with every kernel choice and refusal that GPU makes, and lowers it for CUDA, each of its Pallas kernels to Triton. A kernel that would fail the GPU’s first compile there fails here, in seconds and without the GPU; XLA’s compile of the whole step still needs one. The program exits with status 0 once the step lowers, and opens no W&B run. A TPU topology’s step compiles ahead (compile_ahead).

compile_ahead: Annotated[AheadTarget, tyro.conf.subcommand(ahead)] | None

Compile the run’s first step for GPUs this host lacks, save it, then exit without training.

--trainer.compile-ahead.generation sm90 --trainer.compile-ahead.devices 8 --trainer.compile-ahead.path step.dew-aot builds the run as it would over eight compile-only H100s (dew.training.ahead), compiles its first step there with XLA’s CUDA compiler, which needs the CUDA plugin and no GPU, and writes it with what it costs: each device’s peak bytes and the step’s FLOPs, read before booking the GPUs. The program exits with status 0, and writes no checkpoint or tracking. A TPU topology compiles the same way on any host that holds libtpu, as MaxText’s train_compile does: --trainer.compile-ahead.generation v5e:4x4 --trainer.compile-ahead.devices 16, the step loading on a host of that topology alone.

--trainer.compile-ahead.capacity-gib 80 states each GPU’s memory, and the run’s own fit search then plans the step there as a run would, its log and the saved step naming every program it compiled, each with the bytes derived for it beside XLA’s, and the plan it took, with the seconds planning took. A run with its remat set compiles at that remat alone, and is told of no other; this is how to ask what fits beside it: compile it ahead once at each remat policy its model takes (the model’s remat field) and compare the plans.

ahead: str | None

Run the step compile_ahead saved at this path as the run’s first step, in place of compiling it.

The run lowers its first step as it would and runs the saved executable where that is the same program, on a host of the topology it was compiled for; another program, from another config, data shape, Dew or a plan recorded for another rung, is refused before the run trains.

inspection: Annotated[Inspection, tyro.conf.subcommand(inspect)] | None

What the run watches at its sampled steps (dew.training.inspection.Inspection): parameter paths by glob, the trees summarized there and every how many steps, --trainer.inspection.paths '*norm*' --trainer.inspection.every 100; unset, the step compiles unchanged.

diloco: Annotated[DiLoCo, tyro.conf.subcommand(diloco)] | None

DiLoCo over the mesh’s diloco groups (--trainer.mesh.diloco, dew.training.diloco.DiLoCo): every --trainer.diloco.every windows of updates the groups’ copies are averaged into the outer model, by Nesterov SGD at --trainer.diloco.rate with --trainer.diloco.momentum; unset, one copy trains.

links: Annotated[Mapping[str, Link], record_argument(Mapping[str, Link])]

The links of the run’s split axes, by axis name (Trainer’s links), stated where no process measures them as they are or should not: a step spreads an axis’s projections as its stated link would, --set trainer.links='{"tensor": {"bytes_per_second": 4.5e11, "flops_per_second": 9.9e14}}'. Unnamed, an axis’s link is measured on its devices, and on compile-only ones none is: a step compiled ahead compiles with these and records them (AheadOfTime.links).

tune_kernels: bool

Measure each Pallas launch of the step that has no config measured on this GPU.

Only a process with the GPU to itself measures, and the picks persist beside the compilation cache for every later run. Unset, such a launch runs the nearest measured shape’s config or its heuristic, and the run says how many.

def accumulating(dataset: Loader) -> Self

This config with the accumulation its token budget sets over dataset’s token batches: update_tokens a whole number of them. Raises ValueError for a budget over data batched in rows, one no whole number of them, or an accumulation set otherwise.

def best_policies()
def resumes() -> bool

Whether this run continues from a checkpoint of its own: a named run whose directory holds a step a resume reads (Checkpoints.latest). Its weights then come from that step, so the source it started from need not be read (Pretrained.load’s weights).

def directory(name: str) -> str

The directory of the run name’s checkpoints and record, in checkpoint_dir.

def window(directory: str, name: str) -> ProfileWindow | None

The profile window the run name under directory traces: profile as it names it, or else its own (run_profile) kept with the run’s local records (local_records: its directory’s profile, or Dew’s cache for a run in a bucket) where the process can take a capture (capturable), none where it cannot; none where the run traces none. A trainer is given a window naming its directory alone (row 4246).

def checkpoints(name: str, directory: str | None = None) -> Checkpoints

The checkpoints of the run name in checkpoint_dir (directory), or in directory where given (a saved run read where it lies), keeping keep, with its local tier under name in local_directory, so runs sharing a local disk resume each only its own.

def source_placement(*, frozen: bool) -> SourcePlacement

Where a run that starts from a source (Pretrained.load) reads it: the load’s keywords, frozen where the run keeps the source as it is (an adapter trained over it). A run that resumes reads no weights (resumes), as its checkpoint holds them. A frozen source is read onto this run’s mesh, sharded as its layout shards the state, each process reading its devices’ shards, and the start takes those arrays as they are (Trainer.place): the run and its objective hold one copy. A source the run trains stays on the host, as its start copies the arrays it trains, and so does one whose state lives on the host (Layout.host holds the variables) or keeps its frozen tree whole (frozen_whole), which its start places itself.

def resolved(dataset: Loader) -> tuple[Self, Loader, int | None]

This config and dataset as a run trains on them, and the run’s steps: the accumulation its token budget sets (accumulating), the dataset its batch ramp reads, and the step cap over that (total_steps), before which the ramp goes, since it reads fewer records a step early and a pass over the data is that many steps longer. Its updates are the steps over the accumulation.

def total_steps(dataset: Loader) -> int | None

Return the step cap from steps, epochs or examples, None for a run its tokens bound alone, its own or where its data’s schedule ends (budget_tokens; run_steps, the one policy a direct fit applies too, which refuses a schedule to an objective counting no tokens).

def total_updates(dataset: Loader) -> int | None

Return the optimizer updates the run takes over dataset as RunConfig.train finalizes it, its token budget’s accumulation and its batch ramp applied: what an average that moves once an update ramps over. None for a token-only run.

def eval_interval(dataset: Loader) -> int | str | Counted | None

Return the steps between validation passes over data, or None for never.

def checkpoint_interval(
dataset: Loader,
) -> int | str | Counted | datetime.timedelta | None

Return the steps, or a recorded duration such as 30m, between checkpoints.

function source

def flag_selected(given: Sequence[str], selected: Mapping[str, str]) -> list[str]

given command-line arguments with each optional setting of selected, a field’s name and its subcommand’s, turned on by its own flags: --lora.rank 16 stands for lora:lora --lora.rank 16, where no lora: choice is given. A setting none of whose flags is given stays off.

function source

def looping(
start: Pretrained,
steps: int | None,
layers: tuple[int, int] | None,
*,
backprop_steps: int | None = None,
remat: bool | None = None,
) -> Pretrained

A loaded checkpoint whose block runs steps passes: a looped one’s own block, or layers, which gives a decoder with no loop one over its own layers. Refused where neither names a block, or a new block’s passes are not stated. backprop_steps and remat choose the loop’s backward memory; None keeps a looped checkpoint’s own value.