dew.config
Typed configuration for a training run.
One dataclass tree describes a run: what to build, what to feed it, how to
optimize it, and how the trainer runs. Recipes parse it with tyro, build the
objective and the data it names, and pass both to RunConfig.train, so
to_dict() is a full record of a run and from_dict() rebuilds it.
The model is its class and the fields the run gives it: --model hybrid_dit (an alias or an import path) picks the class, and each of its
own fields is a flag typed by its annotation (--model.num_layers 12,
--model.dtype bfloat16). The data is a chain (dew.data), which the record writes through
the data’s codec (dew.data.records) and the command line sets one setting of at a time:
--set data.<call>.<setting>=... for a call’s (data.token_windows.seq_len=512) and
--set data.source.<field>=... for its store’s.
The resolved config is the run’s spec. A recipe writes it to run.json next
to the checkpoints with save, and load reads it back into the same class,
so inference rebuilds a run from what training was built from. A field the
file lacks takes its declared default, and a field the class does not have
raises an error. The model and the objective record their classes’ defaults
too (ModelConfig.defaults), so a default changed since leaves the record
building what it built.
| Name | Summary |
|---|---|
DecoderRunConfig | A run over a loaded decoder, which may run a block of its layers in passes: the pass count, the block, the passes trained through and their recomputation (CausalTransformer.with_passes), each the run’s choice over the checkpoint’s own. |
MetricConfig | A metric’s class and the arguments the run gives it. |
ObjectiveConfig | An objective’s class and the arguments the run gives it. |
Prepared | What a run trains, as its class builds it (RunConfig.prepare). |
RunConfig | A whole run’s configuration. |
SourceOptions | What a run builds of the source it starts from: Pretrained.load’s keywords for its parts, beside where it reads them (TrainerConfig.source_placement), every run kind’s alike (RunConfig.source_keywords). |
TrainerConfig | Holds the run length, checkpointing, sharding and run tracking. |
flag_selected | given command-line arguments with each optional setting of selected, a field’s name and its subcommand’s, turned on by its own flags: --lora.rank 16 stands for lora:lora --lora.rank 16, where no lora: choice is given. |
looping | A loaded checkpoint whose block runs steps passes: a looped one’s own block, or layers, which gives a decoder with no loop one over its own layers. |
DecoderRunConfig
Section titled “DecoderRunConfig”class DecoderRunConfig( model: ModelConfig = ModelConfig(), data: Annotated[Dataset | None, tyro.conf.Suppress] = None, validation: Annotated[Dataset | Mapping[str, Dataset] | None, tyro.conf.Suppress] = None, data_seed: int = 0, data_plan_version: int | None = None, source: SourceOptions = SourceOptions(), optim: OptimConfig = OptimConfig(), trainer: TrainerConfig = TrainerConfig(), objective: ObjectiveConfig | None = None, lora: Annotated[LoRA, tyro.conf.subcommand(lora)] | None = None, val_metrics: Annotated[tuple[MetricConfig, ...], record_argument(tuple[MetricConfig, ...])] | None = None, val_every: Annotated[Mapping[str, int | str], record_argument(Mapping[str, int | str])] = dict(), val_summaries: Annotated[tuple[Summary, ...], record_argument(tuple[Summary, ...])] = (), loop_steps: int | None = None, loop_layers: tuple[int, int] | None = None, loop_backprop_steps: int | None = None, loop_remat: bool = False,)A run over a loaded decoder, which may run a block of its layers in passes: the pass count, the
block, the passes trained through and their recomputation (CausalTransformer.with_passes), each
the run’s choice over the checkpoint’s own. The language-model and the decision runs take it.
loop_steps: int | None-
The passes a looped checkpoint runs its block (
CausalTransformer.loop), such as Ouro’s at 1, 2 or 4; None keeps the checkpoint’s own. loop_layers: tuple[int, int] | None-
The block of layers [first, end) that
loop_stepsrepeats: a pretrained decoder with no loop gains one over its own layers, as Qwen 3.5’s middle block repeated twice; None keeps a looped checkpoint’s block. loop_backprop_steps: int | None-
Train through the last k passes of the loop alone, with earlier passes cut from backpropagation; None keeps the loop’s own choice.
loop_remat: bool-
Checkpoint every pass of the loop, including the first, and recompute it in the backward pass; requires a loop and leaves decoding unchanged. False keeps a checkpoint’s own choice.
loops: bool-
Whether the run asks anything of a loop (
looped).
DecoderRunConfig.looped
Section titled “DecoderRunConfig.looped”def looped(start: Pretrained) -> Pretrainedstart with its block run as the run asks (looping), as loaded where it asks nothing.
MetricConfig
Section titled “MetricConfig”class MetricConfig( name: str, fields: JsonDict = dict(), defaults: JsonDict = dict(), split: str | None = None,)A metric’s class and the arguments the run gives it.
name is an alias of dew.registry.metrics or an import path, held as
the path. fields are the constructor arguments the run states, as
ObjectiveConfig holds an objective’s: a CLIP metric’s modelname, an
FID’s weights. defaults holds every other argument the class defaults,
so a metric whose default model changes later scores a recorded run as it
did. split is the validation split it scores, every split where None.
MetricConfig.build
Section titled “MetricConfig.build”def build(**derived={}) -> MetricBuild the metric from its arguments and those the run derived
where it states none of its own (the field a video run’s sample
is read from, say).
ObjectiveConfig
Section titled “ObjectiveConfig”class ObjectiveConfig(name: str, fields: JsonDict = dict(), defaults: JsonDict = dict())An objective’s class and the arguments the run gives it.
name is an alias or an import path, held as the path. fields are the
constructor arguments the run states, held as their records
(registry.argument_records), which the command line sets one flag each
(--objective.ema_decay 0.999); what the objective is built around, the
model and what a recipe reads off its data, build takes from the caller.
defaults holds every other argument the class defaults, as ModelConfig
holds a model’s, so an EMA decay, a loss weighting or a sampling step count
the class defaults differently later leaves a recorded run as it was.
ObjectiveConfig.build
Section titled “ObjectiveConfig.build”def build(**derived={}) -> ObjectiveBuild the objective from its arguments and those the caller
derived, which replace a recorded default; an argument the run
states is refused there, since the caller’s would silently replace
it.
Prepared
Section titled “Prepared”class Prepared( run: RunConfig, objective: Objective[LossT, EffectsT], dataset: Dataset | Loader, metrics: Sequence[Metric] = (), rollout: Rollout | None = None, summary: Mapping[str, object] | None = None, validation: Dataset | Mapping[str, Dataset | Split] | None = None, after: Callable[[TrainState, str], None] | None = None,)What a run trains, as its class builds it (RunConfig.prepare).
run is the run as it records itself, with what building resolved (a
pinned source, the model’s fields as built); objective and dataset
what it trains on, and the rest what RunConfig.train takes beside them.
train trains it under a name, and inspected views one sampled update
of it there (RunConfig.viewed).
Prepared.train
Section titled “Prepared.train”def train(name: str) -> TrainStatePrepared.inspected
Section titled “Prepared.inspected”def inspected(name: str) -> dict[str, float]RunConfig
Section titled “RunConfig”class RunConfig( model: ModelConfig = ModelConfig(), data: Annotated[Dataset | None, tyro.conf.Suppress] = None, validation: Annotated[Dataset | Mapping[str, Dataset] | None, tyro.conf.Suppress] = None, data_seed: int = 0, data_plan_version: int | None = None, source: SourceOptions = SourceOptions(), optim: OptimConfig = OptimConfig(), trainer: TrainerConfig = TrainerConfig(), objective: ObjectiveConfig | None = None, lora: Annotated[LoRA, tyro.conf.subcommand(lora)] | None = None, val_metrics: Annotated[tuple[MetricConfig, ...], record_argument(tuple[MetricConfig, ...])] | None = None, val_every: Annotated[Mapping[str, int | str], record_argument(Mapping[str, int | str])] = dict(), val_summaries: Annotated[tuple[Summary, ...], record_argument(tuple[Summary, ...])] = (),)A whole run’s configuration. A kind of run subclasses it with its own
settings and builds what it trains from them (build_run, which prepare calls
on the run with its Hub sources pinned); run trains it.
RESUMABLE: tuple[str, ...]-
The fields a run resumed under its name may change: how it trains and optimizes, what it reads and scores, and, in a kind of run that adds them, how it previews. Every other field says what it trains, which the resume must keep (
check_resume), and so doestrainer.quantization, which changes the forward pass the weights train through. BUILT: tuple[str, ...]-
The fields, by their dotted paths, a run may leave None for its build or its pinning to resolve, whose resolution a later Dew, or a later state of what it reads, may change: the storage of the source it reads (
SourceOptions.param_dtype), the step of a saved Dew run it starts from, which that run’s later checkpoints move (with_pinned_run), the data plan version its order is read by (data_plan_version), and in a kind of run that adds them its own (a decision run’s option text). A run that leaves one None and resumes a saved run under its name takes the saved run’s value (resumed), so running the same command again resumes what it trained; one it sets is its own, and a resume refuses it where it differs (check_resume). A saved run whose record holds one None left it to its build unresolved, which nothing it saved resolves (each earlier shape’s translation resolves what it can,earlier_run), so the value this build resolves is its value. UNSEEDED: str-
How a run of this kind reads a mix saved unseeded and seeded now, as the refusal of its resume names it (
PlanOrderChanged.routed): the data layer’smix(..., seed=None); a decision run’s mixture’s order (dew.decision.data.SCHEDULE_MIXTURE). data: Annotated[Dataset | None, tyro.conf.Suppress]-
The data the run trains on: a chain ending in its preset (
dew.data.presets), whose batch stage is the run’s batch, read and loaded as it says. The record writes it as its chain (dew.data.records); on the command line,--set data.<preset>.<setting>=sets one setting of a recorded chain’s call. validation: Annotated[Dataset | Mapping[str, Dataset] | None, tyro.conf.Suppress]-
The splits each evaluation scores, one dataset or several by name, each batched by its own batch stage or the training batch.
data_seed: int-
The seed of the data’s order and of its random steps.
data_plan_version: int | None-
The data plan version the run reads its order by (
dew.data.plan.PLAN_VERSIONS): None for the saved run’s it resumes, or else this Dew’s, whichresumedresolves on every path a run trains by and its record holds, so a resume reads the order it read whatever a later Dew’s is (BUILT); a saved run that recorded none read by version 8, and one that recorded adata_planof None by 9 (earlier_data_plan). source: SourceOptions-
What the run builds of the source it starts from (
--source.text-only,--source.prediction-depths), where it starts from one (pretrained). objective: ObjectiveConfig | None-
The objective’s class and the arguments the run states, which
trainnames from the objective it trains when the run leaves it unset. lora: Annotated[LoRA, tyro.conf.subcommand(lora)] | None-
The low-rank adapter the run trains instead of the whole model.
On the command line, set it with
--lora.rank 16 --lora.modules q_proj v_proj(seecli). A recipe attaches it to the source that--pretrainedloads (Pretrained.adapt), or, when training from scratch, to a model freshly initialized from the run’s key. The objective then trains only the adapter’s factors, and the run’s record stores the model with the adapter attached. val_metrics: Annotated[tuple[MetricConfig, ...], record_argument(tuple[MetricConfig, ...])] | None-
The metrics each validation split scores, each with the arguments the run gives it (a CLIP metric’s model, an FID’s weights) and the split it scores, every split where it names none (
MetricConfig.split), one JSON list on the command line:--val-metrics '[{"name": "accuracy"}, {"name": "brier", "split": "test"}]'scores accuracy on every split and the Brier score ontest. A split no metric names scores none;[]scores none anywhere. None scores the kind of run’s own on every split (default_metrics): perplexity for a language model, a decision run’s accuracy, calibration, risk-coverage and log loss, a JEPA run’s probes, CLIP for an image run. A metric given is asked for whatever the data holds: a split the run does not validate, or data with no validation split, refuses it. val_every: Annotated[Mapping[str, int | str], record_argument(Mapping[str, int | str])]-
A split’s own validation cadence, as
trainer.eval_everyspells one (EvalSuite.every), one JSON object on the command line:--val-every '{"test": "25%"}'scorestestevery quarter of the run and every other split attrainer.eval_every. A split it names must be one the run validates. val_summaries: Annotated[tuple[Summary, ...], record_argument(tuple[Summary, ...])]-
Scalars weighed from each evaluation’s split scores, a board’s weighted index, each formed at a step that scored every split it weighs and logged as
evaluation/summary/<name>, which--trainer.best evaluation/summary/<name>ranks by (dew.training.evaluation.Summary):--val-summaries '[{"name": "board", "weights": {"val/accuracy": 1, "test/accuracy": 2}, "better": "higher"}]'.
RunConfig.to_dict
Section titled “RunConfig.to_dict”def to_dict() -> dict[str, JSON]Return a JSON-safe record of the run.
A value of a subclass of the field’s type is written as its class’s
import path and its fields. The data and its validation splits are
their chains’ records (dew.data.records.encode), and a chain the
codec cannot describe is {"unrecorded": reason}.
RunConfig.unrecorded
Section titled “RunConfig.unrecorded”def unrecorded() -> list[str]Why each of the run’s chains is not recorded, none where its record can rebuild them all: a run built in Python with a step the record cannot describe trains as built, and its run.json, which says why, refuses to rebuild it.
RunConfig.from_dict
Section titled “RunConfig.from_dict”def from_dict(values: Mapping[str, object]) -> SelfRead back what to_dict wrote, for subclasses too. A field the
record lacks takes its default; an unknown field raises. A chain
recorded as unrecorded, or a recipe’s record written before runs
recorded chains, is refused by name. A model recorded with an earlier
main’s fields reads as its checkpoint’s record does.
RunConfig.override
Section titled “RunConfig.override”def override(point: Mapping[str, JSON]) -> SelfThis config with each dotted path in point replaced, through its record.
A path walks the run’s fields and the fields of the values they hold.
Under model or objective, a name the config does not declare is an
argument of the class it names, as --model.num_layers is on the
command line. A path the run does not declare raises, rather than
training the unchanged config. The paths are set together and the run
is built once, so fields valid only together (a token budget with no
row batch) are one point; each path is placed in this config, so a path
under model names an argument of the model this config holds.
RunConfig.assigned
Section titled “RunConfig.assigned”def assigned(assignments: Sequence[str]) -> SelfThis config with each path=value in assignments set (override).
The value is read as the field’s annotation reads a flag: a number, a string, a literal or a tuple of them as its type parses it on the command line, None for an optional field, and anything else (a record, a list of records) as JSON, or as the text itself when it is not JSON.
RunConfig.save
Section titled “RunConfig.save”def save(directory: str) -> strWrite this config as run.json in directory and return the path.
The file is the run’s class record, its class’s import path and its
fields (to_dict), so load rebuilds the run as the class it is. The
path goes through epath, the same filesystem layer Orbax writes the
checkpoints with, so a gs:// run directory gets the record too.
RunConfig.record
Section titled “RunConfig.record”def record() -> dict[str, JSON]The run’s class record: its class by its alias in dew.registry.runs, or by its import path
where the table holds none, and its fields (to_dict).
RunConfig.source_keywords
Section titled “RunConfig.source_keywords”def source_keywords(*, frozen: bool) -> SourceOptionsKeywordsPretrained.load’s keywords for the source the run starts from: what it builds of it
(source), where it reads it (TrainerConfig.source_placement, frozen where the run
keeps the source as it is) and what it stores its parameters in. Every run kind that starts
from a source loads it with these.
A source the run trains is stored in its numerics’ dtypes.params, the master weights its
updates land in. One it keeps frozen is stored as its checkpoint stores it
(param_dtype='auto'), since no update reaches it: a float32 copy of a bf16 checkpoint doubles
its memory and is cast back to bf16 every pass for nothing, while a cast to the compute dtype
would round the float32 tensors a bf16 checkpoint keeps (Mamba’s A_log). A dtypes.params
narrower than float32 stores it so too. A storage the run’s source options state wins
(SourceOptions.param_dtype).
RunConfig.source_attention
Section titled “RunConfig.source_attention”def source_attention() -> AttentionImplThe attention implementation a run that starts from a source loads it with: its model’s
attention_impl field, auto unless set, refused unless it names one (AttentionImpl). Every run
kind that loads a source reads it here.
RunConfig.built_fresh
Section titled “RunConfig.built_fresh”def built_fresh() -> NoneRefuse source in a run whose model is built from model, loading no source: what it says
Pretrained.load builds of a source (source_keywords) reaches nothing such a run builds.
RunConfig.sourced_model
Section titled “RunConfig.sourced_model”def sourced_model() -> NoneRefuse a model name in a run whose source decides its architecture: such a run reads its
model’s fields’ few choices alone, and a name other than its kind’s own would be dropped.
RunConfig.check_resume
Section titled “RunConfig.check_resume”def check_resume(saved: Mapping[str, object]) -> NoneRefuse to resume the run whose record is saved as this one where they train different
things: another class, or a field outside RESUMABLE that differs. A changed rope_theta
keeps every shape, so it would otherwise continue the old weights under new semantics. The
saved fields are read back as this class reads them (from_dict), so a field the record
predates takes its default, and one it left to its build unresolved takes this build’s.
RunConfig.cli
Section titled “RunConfig.cli”def cli(args: Sequence[str] | None = None, *, default: Self | None = None) -> SelfParse a run from args, or from the process’s own command line when args is None.
An optional setting such as lora is turned on by its own flags. For
example, --lora.rank 16 --lora.modules q_proj stands for
lora:lora --lora.rank 16 ..., and with no --lora. flag the run
trains without an adapter.
--model <alias or import path> picks the model’s class, and every
field the class declares is a flag, --model.<field>, typed by its
annotation (ModelConfig). Naming the class the run defaults to keeps
the default’s fields; naming another starts from that class’s own and
the default’s compute dtype. --objective picks the objective’s class
the same way, each argument it takes a flag, --objective.<argument>
(ObjectiveConfig). --set path=value, once a setting, sets a field at its dotted path
after the flags, as dew train does (assigned): the data’s settings are its chain’s calls’
(--set data.lm.seq_len=512), which have no flags of their own.
RunConfig.load
Section titled “RunConfig.load”def load(directory: str, *, trust: Sequence[str] = ()) -> SelfRead the run a directory’s run.json records, or the file itself,
as the class it names, which must be this one or derive from it.
trust names the packages outside Dew the record may import.
RunConfig.read
Section titled “RunConfig.read”def read(record: Mapping[str, object], *, trust: Sequence[str] = ()) -> SelfThe run a class record (record) names, as its class, which must be
this one or derive from it.
RunConfig.prepare
Section titled “RunConfig.prepare”def prepare() -> PreparedBuild what this run trains, as build_run builds it from the run with the fields it leaves to
its build at the run it resumes (resumed) and its Hub sources pinned (pinned), so the record
names what it started from and is what it loads.
RunConfig.resumed
Section titled “RunConfig.resumed”def resumed(paths: Sequence[str] | None = None) -> SelfThis run with each field it leaves None for its build to resolve (BUILT, or those of them
paths names) at the value the saved run it resumes under its name recorded, read as every
earlier shape of a saved run is (earlier_run), and its data plan version, where it resumes no
run that recorded one, this Dew’s (PLAN_VERSIONS): the one place a run’s version is resolved,
on each path a run trains by (prepare, and readied, which a run trained in Python reaches
alone), before its data compiles or its record is written. Its other fields as they are where it
resumes no run (TrainerConfig.resumes) or leaves none to its build.
RunConfig.build_run
Section titled “RunConfig.build_run”def build_run() -> PreparedBuild what this run trains, the run as it stands, its sources already pinned (prepare
pins them first): the objective its record names around its model, on its data. A kind of
run with settings of its own builds from them in its own build_run.
RunConfig.default_metrics
Section titled “RunConfig.default_metrics”def default_metrics(*, validates: bool) -> tuple[MetricConfig, ...]The metrics a validation pass scores where val_metrics names none, by what the run’s data
validates: a kind of run’s own, none for a run that names its objective alone.
RunConfig.metric
Section titled “RunConfig.metric”def metric(config: MetricConfig) -> Metricconfig built with what this kind of run derives for it where it states none of its own.
RunConfig.validation_metrics
Section titled “RunConfig.validation_metrics”def validation_metrics(*, validates: bool) -> tuple[Metric, ...]The kind of run’s own metrics (default_metrics), built (metric), by what the run’s data
validates: what each split scores where val_metrics names none (suites).
RunConfig.suites
Section titled “RunConfig.suites”def suites( dataset: Loader, validation: Mapping[str, Split] | None, metrics: Sequence[Metric] = (),) -> Mapping[str, EvalSuite] | NoneEach split the run validates, validation’s or dataset.val as val, with the metrics
val_metrics names for it built (metric), else the kind’s own metrics, at its cadence in
val_every, else the run’s; None where neither names any, so each scores the kind’s own at
the run’s. A metric or a cadence for a split the run does not validate is refused.
RunConfig.chain
Section titled “RunConfig.chain”def chain() -> DatasetThe run’s data. Raises ValueError for a run that names none: a run trains on the chain
it states (dew.data.presets), in Python or in its run.json.
RunConfig.compiled
Section titled “RunConfig.compiled”def compiled( objective: ObjectiveData, dataset: Dataset | Loader, validation: Dataset | Mapping[str, Dataset | Entry] | None,) -> tuple[Loader, Mapping[str, Reader | Entry] | None]dataset and validation as the run trains on them (fitted): compiled with objective’s
stages, data_seed and data_plan_version, the training read bounded to trainer.examples where
the run counts them, or to trainer.epochs passes over its source examples where its read has no
record count, so its length is read off the loader this returns.
RunConfig.run
Section titled “RunConfig.run”def run() -> TrainStatePrepare the process, build what this run trains (prepare) and train it.
The run trains under trainer.name, or <objective>-<data>/date-<time>.
RunConfig.pinned
Section titled “RunConfig.pinned”def pinned() -> SelfThis run with its Hub sources named at the commits they resolve to now, as its record names
them and prepare loads them: here an imported adapter’s (LoRA.resolved); a kind of run pins
its own sources beside it. Itself where nothing is left to pin.
RunConfig.with_pinned_run
Section titled “RunConfig.with_pinned_run”def with_pinned_run(pretrained: str | None) -> SelfThis run with the checkpoint it reads of the saved Dew run it starts from (pretrained, a
run’s directory) pinned to the step its selection resolves to now (SourceOptions.step:
best, best:<metric> or its task’s default), so its record names the run and the step it
read and a resume reads them again. Itself where it starts from no saved run, or names a step.
RunConfig.profile
Section titled “RunConfig.profile”def profile( directory: str | None = None, *, warmup: int = 20, steps: int = 10, capture: CaptureOptions | None = None,) -> ProfileReport | NoneTrain this run afresh under directory for warmup steps, then a window of steps more traced
(ProfileWindow), and report where the window’s steps spent their time (ProfileReport), written
as report.json beside XProf’s own reports. The warmup is whole windows, at least one, so the step
compiles before the window and the window is one log interval. The run keeps its own settings but
those that would read or write outside the window, which a profile drops together: its length,
stop, checkpoints local or persistent, the checkpoint ranking (best) and the best model it
would end on, its cooldowns, the validation splits it names and their metrics, cadences and
summaries (a split its chain holds out scores after the window), its W&B sink, and a lowering, a
compile ahead or a step compiled ahead for its own length, which the profile trains past.
directory unset is profiles/<the run’s name>/<the time it starts>, read once the process is
prepared, as run reads a run’s name. The run’s own directory is left as it is. capture is what
the capture records (CaptureOptions), its own by default. Raises ValueError for a directory that
holds this run’s profile already. The report is process zero’s, which logs the run; None on the
others.
RunConfig.recorded
Section titled “RunConfig.recorded”def recorded(objective: Objective[Loss, Effects]) -> SelfReturn this config as the record of the run that trains objective.
The record names the class by its import path, so it says what was
trained and loads in a process that imported nothing. Arguments the
run states belong to the class it names, so a run that states them
for another class than the one it trains is refused. Each argument
the objective resolved from what it was built around
(Objective.resolved) and the run leaves unstated is recorded as
the objective holds it, so the record builds what the run sampled
with.
RunConfig.check_process
Section titled “RunConfig.check_process”def check_process() -> NoneRefuse to train in a process that does not carry the run’s process options
(dew.training.runtime.unprepared): run() and dew train set them up before JAX opens a
backend (prepare_process), and a run trained after prepare() in a process nothing set up
with them would drop them.
RunConfig.readied
Section titled “RunConfig.readied”def readied( objective: Objective[Loss, Effects], dataset: Dataset | Loader, validation: Dataset | Mapping[str, Dataset | Split] | None, metrics: Sequence[Metric], name: str,) -> tuple[Self, Loader, Mapping[str, Split] | None, Mapping[str, EvalSuite] | None, int | None]This run as it trains objective under name, and what it trains on, before anything is
placed (train, viewed): its data compiled with the objective’s own stages and its validation
splits with it, the suites they score (suites), sized and ramped (TrainerConfig.resolved), with
its steps; its trunk quantized where it asks; its record naming the run’s directory and its
optimizer’s groups (grouped).
RunConfig.train
Section titled “RunConfig.train”def train( objective: Objective[Loss, Effects], dataset: Dataset | Loader, *, name: str, metrics: Sequence[Metric] = (), rollout: Rollout | None = None, summary: Mapping[str, object] | None = None, validation: Dataset | Mapping[str, Dataset | Split] | None = None, after: Callable[[TrainState, str], None] | None = None,) -> TrainStateTrain objective on dataset as this config describes; every kind of run calls this after
building both.
The run lives under name in trainer.checkpoint_dir, and process zero writes
the record there before training starts. A trainer.wandb opens a tracker
under the same name, with the record, summary (the recipe’s own view of the
run) and the step count as its config. Only trainer.publish publishes the
final checkpoint to the W&B registry, under the name with slashes and spaces
replaced because an artifact name allows neither. Local tracking journals
live in a separate tracking directory.
dataset is compiled once with the objective’s own stages (dew.data.compile.fitted), and its
validation splits with it; the run’s length, its ramp, its cadences and its recorded plan are
read off that one compiled loader, which fit trains on. Its batch stage is the run’s batch, and
a token budget’s accumulation is trainer.update_tokens over its token batch.
A trainer.quantization wraps the module objective trains before anything
initializes it, so the run learns through the quantized forward pass. A
lora is not applied here: the recipe attaches it to the model and variables
it builds the objective from. rollout is the trainer’s rollout, which turns
each prefetched batch into the one the step trains on, as an on-policy
objective samples it. validation names the splits each evaluation
scores in place of dataset.val, as Trainer.fit takes them: a held-out
split and a test set are logged as val/... and test/.... metrics are
the kind’s own, which each split scores where val_metrics names none. With no
trainer.best, the first split’s loss ranks the checkpoints; a Best
over a metric names its split. after runs on the state training ends
on and the run’s directory before the checkpoint is published, as a
decision run fits its calibration there, so what it writes is published.
RunConfig.compiled_ahead
Section titled “RunConfig.compiled_ahead”def compiled_ahead( objective: Objective[LossT, EffectsT], dataset: Loader, steps: int | None, rollout: Rollout | None, name: str,) -> CompiledAheadThe run’s first step compiled for its compile_ahead target and saved
(AheadTarget.compiled), its trainer built as train builds it, its links the run’s own, over
that target’s compile-only devices (AheadTarget.compile_only), with no tracker: what ends a run
that compiles ahead. A run name whose directory holds its checkpoints reads them, and compiles the
step its resume takes; one that has none writes none.
RunConfig.viewed
Section titled “RunConfig.viewed”def viewed( objective: Objective[Loss, Effects], dataset: Dataset | Loader, *, name: str, rollout: Rollout | None = None, validation: Dataset | Mapping[str, Dataset | Split] | None = None,) -> dict[str, float]The run’s inspection of one sampled update over the batch its first step takes
(dew.training.inspected, first_step_batch), the run readied as train readies it
(readied) and its trainer built as train builds it, from the checkpoints its directory name
holds, their latest, else its start, with no tracker: what inspected returns.
RunConfig.inspected
Section titled “RunConfig.inspected”def inspected() -> dict[str, float]What the run’s inspection (TrainerConfig.inspection) logs of one sampled update over its training
data’s first batch, from its checkpoints’ latest, else its start, offline (viewed): the view its
training logs, for a saved run, which this trains and writes nothing of (dew inspect).
RunConfig.grouped
Section titled “RunConfig.grouped”def grouped(objective: Objective[Loss, Effects], name: str) -> OptimConfigThe run’s optimizer as it trains under name: each group it names that claims the modules its
objective draws fresh beside a loaded base matching those that hold a parameter
(ParamGroup.fresh, trained_modules). Where it names none, a resumed run trains in the groups
its record names, as its optimizer state holds them (none for a record written before runs took
them), and a new run that draws such modules in their default groups (OptimConfig.fresh_groups).
Every process agrees on the groups before any records them.
RunConfig.built
Section titled “RunConfig.built”def built( objective: Objective[LossT, EffectsT], mesh: MeshSpec | jax.sharding.Mesh, rollout: Rollout | None, **resources={},) -> Trainer[LossT, EffectsT]The run’s trainer over mesh: how it steps, every numeric choice its TrainerConfig
holds (key, microbatches, layout, accumulation, split, loss scale, kernel tuning), and the
resources its mode adds (checkpoints, tracker, profile, a lowering or a step compiled
ahead), so a run that trains and one that compiles ahead build the one trainer. The trainer is
the training package’s lazy export (dew.training), read where a run trains or compiles
ahead, so a config read alone reads none of it.
RunConfig.compute_dtype
Section titled “RunConfig.compute_dtype”def compute_dtype() -> strThe dtype the run’s model computes in: the one its record states, else its numerics’
(TrainerConfig.precision).
RunConfig.drawn_dtype
Section titled “RunConfig.drawn_dtype”def drawn_dtype() -> str | NoneThe dtype the run stores the parameters it draws fresh in (Trainer’s param_dtype): its
numerics’ dtypes.params where that is not Flax’s own float32 and its model’s record states
none; None leaves each module’s own.
RunConfig.numerics_fields
Section titled “RunConfig.numerics_fields”def numerics_fields() -> dict[str, str]The fields the run’s numerics derive for its model (ModelConfig.build): its compute dtype,
where the model takes one and its record states none.
RunConfig.sweep
Section titled “RunConfig.sweep”def sweep( space: Space, *, train: Callable[[Self], float], trials: int, ledger: str | Path, tracker: Tracker, search: Search = _RANDOM_SEARCH,) -> list[TrialFinished]Train trials trials of this config over space and return the ledger.
Each trial draws a point from space and trains under the run name
<trainer.name>/trial-<index>, so trials keep their own checkpoints and
tracking, and the score train returns is recorded for it. tracker receives
that score as sweep/value at the trial’s number, along with the trial’s
TrialFinished record. A trial is written to ledger before it is reported,
so rerunning the same call continues an interrupted sweep.
SourceOptions
Section titled “SourceOptions”class SourceOptions( text_only: bool = False, prediction_depths: int | None = None, attention: dict[str, Literal['causal', 'block_bidirectional']] | None = None, dex: Annotated[DifferentialExtension, record_argument(DifferentialExtension)] | None = None, scan_layers: bool = False, param_dtype: str | None = None, step: int | str | None = None, ema: bool | None = None,)What a run builds of the source it starts from: Pretrained.load’s keywords for its parts,
beside where it reads them (TrainerConfig.source_placement), every run kind’s alike
(RunConfig.source_keywords).
text_only: bool-
A multimodal wrapper’s language model alone: no tower, projector or audio encoder built, nor their tensors read (
Pretrained.load’stext_only). prediction_depths: int | None-
The first that many multi-token prediction depths, 0 for none; None the checkpoint’s own (
Pretrained.load’sprediction_depths). attention: dict[str, Literal['causal', 'block_bidirectional']] | None-
How each layer kind’s attention reads, by kind (
full_attention): causal, or block_bidirectional, each of a row’s blocks read whole, which a run’s rows then lie in (Pretrained.load’smodeling_overrides); None the checkpoint’s own:--source.attention full_attention block_bidirectional. dex: Annotated[DifferentialExtension, record_argument(DifferentialExtension)] | None-
DEX’s differential on chosen heads (
dew.nn.dex,Pretrained.load’sdex), its record as JSON:--source.dex '{"layers": [[0, [1, 3]], [5, [0, 2]]], "anneal_steps": 1000}', itstrainsthe paper’s recipe unless “all”; None for none. scan_layers: bool-
The decoder’s runs of like layers, and of layers repeating one period, under flax’s scan (
Pretrained.load’sscan_layers): a decode step’s program grows with the period, not the depth. param_dtype: str | None-
The dtype the source’s parameters are stored in, its checkpoint’s own as ‘auto’; None stores a source the run trains in its numerics’
dtypes.paramsand one it keeps frozen as its checkpoint does (RunConfig.source_keywords). A frozen bfloat16 base beside a float32 head and adapter is--source.param-dtype bfloat16. step: int | str | None-
Where the run starts from a saved Dew run (
--pretrained <run directory>), the checkpoint it reads: a step,latest,best,best:<metric>, or None for the one the run’s task loads by default (dew.interop.saved_sources.released_run). The run records the step it read (RunConfig.pinned). ema: bool | None-
Where the run starts from a saved Dew run, whether it reads that run’s EMA weights; None its task’s own choice (
SavedRun.variables).
SourceOptions.overrides
Section titled “SourceOptions.overrides”def overrides() -> Mapping[str, object] | NoneThe modeling_overrides these options state, None where they state none.
SourceOptions.keywords
Section titled “SourceOptions.keywords”def keywords() -> SourceBuildPretrained.load’s keywords for what a run builds of its source. With weights=False beside
them a load is the run’s base as the run builds it, its config and its variables’ shapes, read
from the source’s config and safetensors headers alone, no weight downloaded (a release of
PyTorch pickles alone, whose shapes no header states, is read whole): a check that a base
loads under a run’s options, on a disk too small for its weights.
TrainerConfig
Section titled “TrainerConfig”class TrainerConfig( name: str | None = None, checkpoint_dir: str = './checkpoints', keep: Annotated[int | Keep, _keep_argument()] = Keep.latest, resume: Resume = 'auto', best: Annotated[str | Best | tuple[Best, ...] | None, _best_argument()] = None, update_tokens: int | None = None, key: int = 0, steps: int | None = None, epochs: int | None = None, tokens: int | None = None, examples: int | None = None, stop: Annotated[Pause, tyro.conf.subcommand(pause)] | Annotated[Plateau, tyro.conf.subcommand(plateau)] | None = None, restore_best: bool = False, cooldowns: Annotated[Cooldowns, tyro.conf.subcommand(cooldowns)] | None = None, log_every: Annotated[int | str, _cadence_argument(timed=False)] = 100, eval_every: Annotated[int | str | None, _cadence_argument(timed=False)] = 'epoch', checkpoint_every: Annotated[int | str | datetime.timedelta | None, _cadence_argument()] = 'epoch', checkpoint_final: bool = True, local_directory: str | None = None, local_every: int | str | None = None, scored_to: str | None = None, accumulation: int = 1, split: int | None = None, batch_ramp: Ramp | None = None, dynamic_scale: bool | None = None, precision: Annotated[Numerics, _numerics_argument()] = Numerics(), mesh: MeshSpec = MeshSpec(), microbatches: int | None = None, layout: Layout = Layout(), profile: ProfileWindow | None = None, run_profile: RunProfile | None = RunProfile(), compilation_cache_dir: str | None = None, wandb: Wandb | None = None, publish: bool = False, multi_host: bool | None = None, xla_flags: str | None = None, quantization: Quantization | None = None, lower_for: str | None = None, compile_ahead: Annotated[AheadTarget, tyro.conf.subcommand(ahead)] | None = None, ahead: str | None = None, inspection: Annotated[Inspection, tyro.conf.subcommand(inspect)] | None = None, diloco: Annotated[DiLoCo, tyro.conf.subcommand(diloco)] | None = None, links: Annotated[Mapping[str, Link], record_argument(Mapping[str, Link])] = dict(), tune_kernels: bool = False,)Holds the run length, checkpointing, sharding and run tracking.
keep: Annotated[int | Keep, _keep_argument()]-
The number of latest checkpoints kept, besides the best one (
Keep’s own default). resume: Resume-
Whether the run continues its directory’s latest checkpoint: where there is one, “require” for a job that must not start afresh, or “never” for one that must not continue another (
Trainer.fit). best: Annotated[str | Best | tuple[Best, ...] | None, _best_argument()]-
The metric name and ranking policy; None selects validation loss or training loss.
update_tokens: int | None-
The optimizer update’s token budget, a whole number of the data’s token batches (its batch stage’s
TokenBudget), which sets the run’s accumulation (RunConfig.train). key: int-
The seed of the run key, which sets parameter init and every per-step draw.
epochs: int | None-
The run length as passes over the data;
stepssets it directly instead. tokens: int | None-
Stop after the loss has scored this many targets across all microbatches, including before a resume.
Counts the tokens committed updates trained on, reported as
train/committed_tokens, without padding. The final microbatch may cross the budget. Withsteps,epochsorexamples, the first cap reached ends the run. A token-only run needs a constant learning rate or a schedule with its own explicit step horizon. examples: int | None-
The run length as the first this many source examples of the data, each read once with every row it makes: the records the chain makes in its own order, counted before they are packed and before an objective makes several rows of one (a decision layout’s row a question) (
Trainer.fit’sexamples);stepsorepochsset it instead. stop: Annotated[Pause, tyro.conf.subcommand(pause)] | Annotated[Plateau, tyro.conf.subcommand(plateau)] | None-
End the run at a fraction of its length, to resume it later as if it had never stopped (
Pause):--trainer.stop.fraction 0.5; or once a metric, by its recorded name asbestnames one, stops improving (Plateau):trainer.stop:plateau --trainer.stop.metric val/loss --trainer.stop.evals 3. restore_best: bool-
End on the model the run ranked best (
Trainer.fit’srestore_best): the state the run hands on (RunConfig.train’safter) holds the best step’s weights, EMA and counters, with the final step’s optimizer state. The checkpoints stay as written;dew export --step bestexports the best one. cooldowns: Annotated[Cooldowns, tyro.conf.subcommand(cooldowns)] | None-
Branch the run into cooldowns, each scored by its validation and discarded (
Cooldowns):--trainer.cooldowns.at 20% 40% --trainer.cooldowns.length 20%. log_every: Annotated[int | str, _cadence_argument(timed=False)]-
The steps between logs, or a share of the run such as “1%” (
eval_every). eval_every: Annotated[int | str | None, _cadence_argument(timed=False)]-
The steps between validation passes.
It is a number of steps, “epoch” for one pass over the data, a share of the run such as “5%”, resolved once against its planned length in steps, epochs, examples or tokens (
dew.training.Trainer.fit), or None to never validate. “epoch” over a stream that reports no record count raises a ValueError, since such a stream has no pass. checkpoint_every: Annotated[int | str | datetime.timedelta | None, _cadence_argument()]-
The steps between checkpoints, with the same choices as
eval_every, or a duration such as 30m.A stream whose iterator cannot report its read position must train with None; the trainer refuses any other value for it.
checkpoint_final: bool-
Write the final state when training ends, independently of
checkpoint_every.False disables this save; cadenced, ranked and preemption saves keep their own policy.
local_directory: str | None-
A path on every host’s own disk where the run also writes checkpoints every
local_everysteps, under its name as incheckpoint_dir, cheaper than the persistent ones and lost with the host (Checkpoints); a resume reads the newer of the two. Set withlocal_every, or neither. local_every: int | str | None-
The steps between local checkpoints, or a share of the run such as “5%” (
local_directory). scored_to: str | None-
A directory that takes every validation batch’s scoring artifacts at each evaluation, a decision run’s decisions among them, as
<scored_to>/step-<step>/<split>-<batch>.npz, for pooling a pass’s metrics offline (Trainer.fit’sscored_to); None writes none. accumulation: int-
The number of steps per optimizer update.
split: int | None-
The microbatches each step’s batch is read in, its rows split evenly.
None lets the trainer plan it from the compiled step’s memory: the fewest that fit the devices before it recomputes anything, at the least recomputation that fits (
Trainer.compile). A step’s tokens and the update’s stay as configured either way; only the microbatch shrinks. batch_ramp: Ramp | None-
A ramp that grows
batch_sizeover the run’s first records instead of starting there.It sets the global batch the run starts at, what each stage adds, and the records the whole ramp spans. Unset trains at
batch_sizethroughout. Either way there is one optimizer update a step, and the compiled step is traced once per stage. dynamic_scale: bool | None-
Whether the run scales its loss dynamically (
flax.training.dynamic_scale.DynamicScale): None for a float16 compute’s alone, which needs it (RunConfig.built); a run states True or False to choose, as a run saved before its numerics records the one it trained under. precision: Annotated[Numerics, _numerics_argument()]-
The run’s numerics, set once and recorded with it (
dew.nn.precision.Numerics;ACCURATEfor scientific runs): its dtypes by Lightning’s names, its products’ precision by JAX’s levels and its ops’ forms, over every program it traces, the step and its evaluations (dew.nn.kernels.generation.run_precision). Its compute dtype is the model’s where the model’s record states none (RunConfig.numerics_fields), its parameters’ a loaded checkpoint’s storage, and a float16 compute scales the loss (RunConfig.built). The optimizer’s moments are a storage format of their own (OptimConfig.state_dtype: float32 under any weights, bfloat16 stochastically rounded, int8 blockwise). A dtype or precision the model states still wins, and a resumed run traces under the numerics it records. microbatches: int | None-
Microbatches a pipelined step feeds through the mesh’s stages, a multiple of
mesh.stage; unset feeds one a stage. profile: ProfileWindow | None-
One profiler window: the steps to trace, the warmup before them and the output directory, in place of the run’s own (
run_profile). Unset traces the run’s own window alone. run_profile: RunProfile | None-
The run’s own profile window, every run’s by default: two of its steps traced once ten have run past its first compile, kept with its records (
window), its trace and manifest alone, where the process can take a capture (dew.training.profiling.capturable): a capture’s fixed cost, about a second on a CPU host whatever its steps (row 4112), a run of minutes does not see, and a run shorter than its warmup takes none. Aprofilethe run names replaces it; None traces none of its own. compilation_cache_dir: str | None-
The directory of JAX’s persistent compilation cache, so a restart skips recompiling the step.
None caches under the user’s cache directory wherever the run’s processes share it (
dew.training.runtime.ready_process); a pool across hosts without shared storage names one they share. JAX_ENABLE_COMPILATION_CACHE=false compiles from scratch every run. wandb: Wandb | None-
An optional W&B telemetry sink, in addition to the local tracking journal.
Scalars, records and objective previews are tracked; checkpoints are not uploaded.
publish: bool-
Publish the final checkpoint to W&B; requires
wandb, online, andcheckpoint_final. multi_host: bool | None-
Whether to join the JAX process pool.
None tries, and continues alone only when no cluster is configured; True requires the pool; False never joins.
xla_flags: str | None-
Extra XLA_FLAGS for this run, appended to the environment by
prepare_process.dew trainsets them in XLA_FLAGS before the process loads jaxlib, so they hold for every runtime XLA loads (dew._devices.hold_xla_flags);prepare_processappends them before JAX opens a backend, unproven where jaxlib has loaded. Library users set XLA_FLAGS before the process starts to make them certain; docs/performance.md records what was measured. quantization: Quantization | None-
The quantized-training spec, wrapped around the module the objective trains.
The wrapper is applied before the run initializes the module. Unset trains in the compute dtype.
dew.quantizationdescribes what the wrapper does and what it keeps. lower_for: str | None-
Check the run’s step for a GPU on any host, a CPU too, then exit without training.
--trainer.lower-for sm80(or sm75, sm86, sm89, sm90, sm120) builds the run as it would, traces its first step as on a GPU of that CUDA generation, with every kernel choice and refusal that GPU makes, and lowers it for CUDA, each of its Pallas kernels to Triton. A kernel that would fail the GPU’s first compile there fails here, in seconds and without the GPU; XLA’s compile of the whole step still needs one. The program exits with status 0 once the step lowers, and opens no W&B run. A TPU topology’s step compiles ahead (compile_ahead). compile_ahead: Annotated[AheadTarget, tyro.conf.subcommand(ahead)] | None-
Compile the run’s first step for GPUs this host lacks, save it, then exit without training.
--trainer.compile-ahead.generation sm90 --trainer.compile-ahead.devices 8 --trainer.compile-ahead.path step.dew-aotbuilds the run as it would over eight compile-only H100s (dew.training.ahead), compiles its first step there with XLA’s CUDA compiler, which needs the CUDA plugin and no GPU, and writes it with what it costs: each device’s peak bytes and the step’s FLOPs, read before booking the GPUs. The program exits with status 0, and writes no checkpoint or tracking. A TPU topology compiles the same way on any host that holds libtpu, as MaxText’s train_compile does:--trainer.compile-ahead.generation v5e:4x4 --trainer.compile-ahead.devices 16, the step loading on a host of that topology alone.--trainer.compile-ahead.capacity-gib 80states each GPU’s memory, and the run’s own fit search then plans the step there as a run would, its log and the saved step naming every program it compiled, each with the bytes derived for it beside XLA’s, and the plan it took, with the seconds planning took. A run with its remat set compiles at that remat alone, and is told of no other; this is how to ask what fits beside it: compile it ahead once at each remat policy its model takes (the model’srematfield) and compare the plans. ahead: str | None-
Run the step
compile_aheadsaved at this path as the run’s first step, in place of compiling it.The run lowers its first step as it would and runs the saved executable where that is the same program, on a host of the topology it was compiled for; another program, from another config, data shape, Dew or a plan recorded for another rung, is refused before the run trains.
inspection: Annotated[Inspection, tyro.conf.subcommand(inspect)] | None-
What the run watches at its sampled steps (
dew.training.inspection.Inspection): parameter paths by glob, the trees summarized there and every how many steps,--trainer.inspection.paths '*norm*' --trainer.inspection.every 100; unset, the step compiles unchanged. diloco: Annotated[DiLoCo, tyro.conf.subcommand(diloco)] | None-
DiLoCo over the mesh’s diloco groups (
--trainer.mesh.diloco,dew.training.diloco.DiLoCo): every--trainer.diloco.everywindows of updates the groups’ copies are averaged into the outer model, by Nesterov SGD at--trainer.diloco.ratewith--trainer.diloco.momentum; unset, one copy trains. links: Annotated[Mapping[str, Link], record_argument(Mapping[str, Link])]-
The links of the run’s split axes, by axis name (
Trainer’slinks), stated where no process measures them as they are or should not: a step spreads an axis’s projections as its stated link would,--set trainer.links='{"tensor": {"bytes_per_second": 4.5e11, "flops_per_second": 9.9e14}}'. Unnamed, an axis’s link is measured on its devices, and on compile-only ones none is: a step compiled ahead compiles with these and records them (AheadOfTime.links). tune_kernels: bool-
Measure each Pallas launch of the step that has no config measured on this GPU.
Only a process with the GPU to itself measures, and the picks persist beside the compilation cache for every later run. Unset, such a launch runs the nearest measured shape’s config or its heuristic, and the run says how many.
TrainerConfig.accumulating
Section titled “TrainerConfig.accumulating”def accumulating(dataset: Loader) -> SelfThis config with the accumulation its token budget sets over dataset’s token batches:
update_tokens a whole number of them. Raises ValueError for a budget over data batched in
rows, one no whole number of them, or an accumulation set otherwise.
TrainerConfig.best_policies
Section titled “TrainerConfig.best_policies”def best_policies()TrainerConfig.resumes
Section titled “TrainerConfig.resumes”def resumes() -> boolWhether this run continues from a checkpoint of its own: a named run whose directory holds a
step a resume reads (Checkpoints.latest). Its weights then come from that step, so the source
it started from need not be read (Pretrained.load’s weights).
TrainerConfig.directory
Section titled “TrainerConfig.directory”def directory(name: str) -> strThe directory of the run name’s checkpoints and record, in checkpoint_dir.
TrainerConfig.window
Section titled “TrainerConfig.window”def window(directory: str, name: str) -> ProfileWindow | NoneThe profile window the run name under directory traces: profile as it names it, or else its
own (run_profile) kept with the run’s local records (local_records: its directory’s profile,
or Dew’s cache for a run in a bucket) where the process can take a capture (capturable), none
where it cannot; none where the run traces none. A trainer is given a window naming its directory
alone (row 4246).
TrainerConfig.checkpoints
Section titled “TrainerConfig.checkpoints”def checkpoints(name: str, directory: str | None = None) -> CheckpointsThe checkpoints of the run name in checkpoint_dir (directory), or in directory where
given (a saved run read where it lies), keeping keep, with its local tier under name in
local_directory, so runs sharing a local disk resume each only its own.
TrainerConfig.source_placement
Section titled “TrainerConfig.source_placement”def source_placement(*, frozen: bool) -> SourcePlacementWhere a run that starts from a source (Pretrained.load) reads it: the load’s keywords,
frozen where the run keeps the source as it is (an adapter trained over it). A run that
resumes reads no weights (resumes), as its checkpoint holds them. A frozen source is read onto
this run’s mesh, sharded as its layout shards the state, each process reading its devices’
shards, and the start takes those arrays as they are (Trainer.place): the run and its
objective hold one copy. A source the run trains stays on the host, as its start copies the
arrays it trains, and so does one whose state lives on the host (Layout.host holds the
variables) or keeps its frozen tree whole (frozen_whole), which its start places itself.
TrainerConfig.resolved
Section titled “TrainerConfig.resolved”def resolved(dataset: Loader) -> tuple[Self, Loader, int | None]This config and dataset as a run trains on them, and the run’s steps: the accumulation
its token budget sets (accumulating), the dataset its batch ramp reads, and the step cap over
that (total_steps), before which the ramp goes, since it reads fewer records a step early
and a pass over the data is that many steps longer. Its updates are the steps over the
accumulation.
TrainerConfig.total_steps
Section titled “TrainerConfig.total_steps”def total_steps(dataset: Loader) -> int | NoneReturn the step cap from steps, epochs or examples, None for a run its tokens bound
alone, its own or where its data’s schedule ends (budget_tokens; run_steps, the one policy
a direct fit applies too, which refuses a schedule to an objective counting no tokens).
TrainerConfig.total_updates
Section titled “TrainerConfig.total_updates”def total_updates(dataset: Loader) -> int | NoneReturn the optimizer updates the run takes over dataset as RunConfig.train finalizes it,
its token budget’s accumulation and its batch ramp applied: what an average that moves once an
update ramps over. None for a token-only run.
TrainerConfig.eval_interval
Section titled “TrainerConfig.eval_interval”def eval_interval(dataset: Loader) -> int | str | Counted | NoneReturn the steps between validation passes over data, or None for never.
TrainerConfig.checkpoint_interval
Section titled “TrainerConfig.checkpoint_interval”def checkpoint_interval( dataset: Loader,) -> int | str | Counted | datetime.timedelta | NoneReturn the steps, or a recorded duration such as 30m, between checkpoints.
flag_selected
Section titled “flag_selected”def flag_selected(given: Sequence[str], selected: Mapping[str, str]) -> list[str]given command-line arguments with each optional setting of selected, a field’s name and its
subcommand’s, turned on by its own flags: --lora.rank 16 stands for lora:lora --lora.rank 16,
where no lora: choice is given. A setting none of whose flags is given stays off.
looping
Section titled “looping”def looping( start: Pretrained, steps: int | None, layers: tuple[int, int] | None, *, backprop_steps: int | None = None, remat: bool | None = None,) -> PretrainedA loaded checkpoint whose block runs steps passes: a looped one’s own block, or layers,
which gives a decoder with no loop one over its own layers. Refused where neither names a block,
or a new block’s passes are not stated. backprop_steps and remat choose
the loop’s backward memory; None keeps a looped checkpoint’s own value.