dew.training
The trainer and the pieces it is built from.
dew.training contains nothing specific to a modality. It imports nothing from
dew.diffusion, dew.inputs or dew.sampling at runtime, and imports wandb
only when a WandbTracker logs.
Each name is imported from its own module when it is first used (dew._exports.lookup), so
import dew.training.budget reads that module, not the trainer.
| Name | Summary |
|---|---|
Aux | Everything a loss returns besides its statistics. Documented in dew.objectives.base. |
Best | Keeps the top best evaluations by a metric, a recorded metric name, or a custom score. |
CPUOnlyJax | A machine with an NVIDIA GPU whose JAX opened with none (refuse_cpu_only_jax). |
Checkpoints | Manages the checkpoints of one run in one directory. Documented in dew. |
Cooldowns | Where a run branches into cooldowns, and how each decays. Documented in dew.training.cooldown. |
DiLoCo | DiLoCo’s schedule and outer optimizer: the groups meet every every windows of updates, and the outer step is Nesterov SGD of the outer gradient at rate with momentum, 0.7 and 0.9 by default (the paper’s, MaxText’s diloco_outer_lr and diloco_outer_momentum). |
EMASpec | Which leaves the EMA copy tracks, and how fast it follows them. Documented in dew.objectives.base. |
Episodes | A split scored by flying a policy, not by reading records: score(variables, key) returns the split’s scores by name (dew.rl.evaluation.episodes), each declared with how it shows (scores, the split’s metrics, EpisodeScore). |
EvalSuite | A validation split fit scores with its own metrics on its own cadence. |
Evaluation | The result of one evaluation, with its previews kept on process 0. |
Inspection | What a run watches and how often (row 2317). |
Keep | Keeps the union of the latest steps, periodic steps, wall-time-spaced steps and a predicate’s steps. Documented in dew. |
Layout | Decides how a train state is placed on a mesh. Documented in dew.distributed.placement. |
LocalTracker | Writes each report to a tracking directory as it is logged. |
MLflowTracker | An MLflow run in experiment, opened on the first value logged into it. |
MeshSpec | Device counts for the mesh’s sharding axes; the data axis takes the devices left over. Documented in dew.distributed.placement. |
Metric | Reduces a validation pass to one scalar, on the host. Documented in dew.objectives.base. |
Objective | Defines what is learned: the parameters, the loss and what evaluation produces. Documented in dew.objectives.base. |
Observe | call(state) at every every-th step of a fit and at its last, its result kept in values. |
PassesProgress | A custom training step (Trainer(step=)) that passes the run’s progress to a model that reads it (dew.nn.progress, Progress(state.updates)), as the objective’s own loss does: make, its factory, wrapped. |
Pause | Ends the run once it has trained fraction of its budget, its state and data position written in full, so a run resumed from that checkpoint trains on as one that never paused: the budget still sizes the learning-rate schedule. Documented in dew.training.budget. |
Plateau | Stops training after evals eligible evaluations without an improvement larger than min_delta. Documented in dew.training.budget. |
Preempted | Raised by Trainer.fit when it stops at a preemption notice. Documented in dew.training.runtime. |
ProfileWindow | A request for one profiler window per fit. |
Rollout | Produces a batch on the host, before the compiled step reads it. |
RunProfile | A run’s own profile window, every run’s by default (dew.config.TrainerConfig.run_profile): steps traced once warmup have run past its first compile, kept with the run’s records, its trace and manifest alone (no reports: no XProf asked, no conversion the run’s end waits for), and taken where the process can take one (capturable), none where it cannot, where a ProfileWindow a run names refuses. |
Step | What the trainer tells an objective about the current step. Documented in dew.objectives.base. |
TensorBoardTracker | A TensorBoard event file in directory, opened on the first value logged into it. |
Tracker | |
Trackers | Sends each report to several trackers. |
Trainer | Runs an Objective: gradients, sharding, EMA, checkpoints and logging. |
Wandb | Where a run reports to Weights & Biases. |
WandbTracker | A Weights & Biases run, opened on the first value logged into it. |
ema_update | Update the EMA leaves toward params, keeping each leaf’s storage dtype. Documented in dew.averaging. |
inspected | The scalars trainer’s inspection logs, of one sampled update over batch from the state the trainer places, its checkpoints’ latest where it has them, else its start: the view a live run logs, for a loaded run on one batch offline. |
prepare_process | Set the environment variables and XLA flags, raise the soft file descriptor limit, and join the JAX process pool. Documented in dew.training.runtime. |
run_timestamp | Return process 0’s wall clock as %Y-%m-%d_%H:%M:%S, on every process. Documented in dew.training.runtime. |
class Best( metric: str | Metric | TrainingScalar | Summary | types.MethodType | Callable[[Mapping], float], top: int = 1, mode: Literal['min', 'max'] | None = None, threshold: float | None = None, weights_only: bool = False, split: str | None = None, *, _source: Metric | TrainingScalar | Summary | Callable[[Mapping], float] | None = None,)Keeps the top best evaluations by a metric, a recorded metric name, or a custom score.
A name refers to one of the metric objects passed to fit, or to a Summary as
evaluation/summary/<name>. mode sets the direction, ‘min’ or ‘max’. When it is None, a custom
score is minimized, and a named metric or a metric object takes its direction
from its Shown declaration. An evaluation whose score is not better
than threshold is not ranked.
Best.resolved
Section titled “Best.resolved”def resolved(scored: Mapping[str, Sequence[Metric | Summary]]) -> BestThis selection bound to the split it ranks and the metric it names there: train for a
training scalar, else the split of scored (each split’s metrics) it names, by split or as
its recorded name’s prefix, or the only one, and the metric it names among that split’s own,
so each suite’s instance of one metric is its own. Its direction, where it names none, is how
that metric itself declares it is shown.
CPUOnlyJax
Section titled “CPUOnlyJax”class CPUOnlyJax(RuntimeError)A machine with an NVIDIA GPU whose JAX opened with none (refuse_cpu_only_jax).
DiLoCo
Section titled “DiLoCo”class DiLoCo(every: int, rate: float = 0.7, momentum: float = 0.9)DiLoCo’s schedule and outer optimizer: the groups meet every every windows of updates, and the
outer step is Nesterov SGD of the outer gradient at rate with momentum, 0.7 and 0.9 by default (the
paper’s, MaxText’s diloco_outer_lr and diloco_outer_momentum). The groups are the mesh’s diloco
axis (MeshSpec.diloco).
every: int-
Windows of updates between the groups’ meetings (MaxText’s
diloco_sync_period). outer: optax.GradientTransformation-
The outer optimizer: Nesterov SGD at
ratewithmomentum.
DiLoCo.admits
Section titled “DiLoCo.admits”def admits(host: Sequence[str], record: bool) -> NoneRefuse what a DiLoCo run cannot express, before any state is made or supplied: a layout that
keeps fields on the host (host, Layout.host), whose offload and host-master step place the
outer model’s fields and run no group’s, and a record of each example’s draws (record), which
no single tape holds of several groups’ own draws.
DiLoCo.started
Section titled “DiLoCo.started”def started(state: TrainState, mesh: Mesh) -> TrainStateThe state a DiLoCo run over mesh’s groups starts from: each group’s inner state a copy of
state’s, the outer variables state’s with their parameters in float32, the outer optimizer’s
state fresh.
DiLoCo.step
Section titled “DiLoCo.step”def step(body: Callable, mesh: Mesh, split: int) -> Callablebody, a step over one group’s state and rows (Trainer._step_body), run by every group on its
own share of the batch under jax.shard_map over the diloco axis alone, then the groups met where
the step closes their everyth window (synced). Its loss, floating metrics and commit telemetry
are the groups’ mean, the rest the first group’s (_agreed): its commits’ count and tokens, which
each group counts of the whole batch.
DiLoCo.synced
Section titled “DiLoCo.synced”def synced(state: TrainState) -> TrainStatestate with its groups met: the outer gradient the outer parameters less the groups’ mean,
stepped by the outer optimizer in float32, every group’s parameters the outer ones again; the
other collections and the EMA the groups’ mean where they are floating, the first group’s
otherwise. Each group keeps its inner optimizer’s state.
Episodes
Section titled “Episodes”class Episodes( score: Callable[[Variables, jax.Array], Mapping[str, float]], scores: Mapping[str, Shown],)A split scored by flying a policy, not by reading records: score(variables, key) returns the
split’s scores by name (dew.rl.evaluation.episodes), each declared with how it shows (scores,
the split’s metrics, EpisodeScore). It goes where a reader goes (EvalSuite(Episodes(...))), and
Evaluation.run scores it on the averaged weights where the run keeps them, on every process and
agreed (a scorer that fails on one, or reports other scores than scores declares, fails the
evaluation on all), process 0’s scores the pool’s, each of the metrics asked for (all of metrics
unless a suite names fewer) named <split>/<name> as a data split’s; a warming pass flies to compile
and keeps none. Its key is
the evaluation’s event key (_event: the run’s key folded with ‘EVAL’ and the step) folded with
‘EPIS’. It reads no batches, so a run’s data metrics and loss are not its.
metrics: tuple[Metric, ...]-
The split’s metrics: one
EpisodeScorea declared score, made once, so a selection given one (Best(split.metrics[0])) reads the very metric the run declares.
EvalSuite
Section titled “EvalSuite”class EvalSuite( data: Split, metrics: Sequence[Metric] = (), every: int | str | Counted | None = None,)A validation split fit scores with its own metrics on its own cadence.
Empty metrics scores fit’s, and every None scores it on fit’s
eval_every. Every suite is also scored at a checkpoint that ranks the
run’s steps and at the end, so no ranking reads an earlier score. A split
scored by its episodes holds an Episodes as its data.
Evaluation
Section titled “Evaluation”class Evaluation( step: int, split: str, scores: dict[str, float], coordinated_batches: int, records: int, event_key: tuple[int, ...], elapsed_seconds: float, previews: tuple[Artifact, ...], examples: int | None = None, real_token_share: float | None = None,)The result of one evaluation, with its previews kept on process 0.
Every process holds the same scores, row counts and RNG identity
(event_key). Each score’s name starts with the split, as in val/loss.
coordinated_batches is the number of batches every process scored, and
records the number of real rows in them, which covers every record of the
split once. elapsed_seconds is process 0’s wall time, including closing the
batch iterator. examples is the source examples the run had consumed when it
evaluated, on the examples clock, its own coordinate there; None on another. The
metric accumulators and the validation batches are not kept.
real_token_share: float | None-
Of the tokens the split’s batches computed (each row’s
Objective.row_tokens, padding and a drained rank’s repeats included), the share the loss scored (Objective.scored_tokensof its real rows); None where the objective counts no tokens or its rows declare none. scalars: dict[str, float]-
Return the metric values, plus the split’s counts if a batch was scored (
evaluation/counts/<split>/records, each split’s its own at a step that scores several, apart from every summary’s,Summary.SPLIT, and itsreal_token_sharewhere known), and on the examples clock the examples the run had consumed (EXAMPLES_SCALAR), a cooldown branch’s under its own name (COOLDOWN).
Evaluation.branched
Section titled “Evaluation.branched”def branched() -> EvaluationThis evaluation as a cooldown branch’s: its split and every score named under COOLDOWN.
Evaluation.run
Section titled “Evaluation.run”def run( objective: Objective[Loss, Effects], variables: Variables, batches: Reader | Dataset | Episodes | None, *, key: int | jax.Array, metrics: Sequence[Metric] = (), step: int | jax.Array = 0, averaged: Variables | None = None, preview: bool = False, mesh: Mesh | None = None, split: str = 'val', schedule_step: int | jax.Array | None = None, loss: bool = False, training: Mapping[str, jax.Array] | None = None, warm: bool = False, updates: int | jax.Array | None = None, scored_to: str | os.PathLike[str] | None = None,) -> EvaluationEvaluate variables on the batches every process can read, without an optimizer or a tracker.
Every process calls this with the same objective, metrics and numerical
settings. batches is called with this process’s share of the split
(DataPartition.of(mesh)) and returns a fresh iterator, which this call
opens and closes; you can pass Loader.val directly, or a chain
(Dataset), which compiles here with the objective’s own evaluation
stages and reads in its own batches (dew.data.compile.held). Every batch holds
VALID_ROWS (all True when the reader marked none), and the objective’s loss
and the metrics count only the real rows. A process whose share runs out
before the others’ scores a copy of its last batch in which every row is a
repeat, so every process scores the same number of batches and the pass
reads every record of every share. With no metrics, no loss and no
preview, no iterator is opened and the objective does no work, and a preview
alone reads at most the first batch. Only process 0’s preview flag counts,
and the preview is made once per call.
variables is the complete Flax variables tree. averaged, when given, is
the complete set of averaged weights the objective sees as Step.ema, not an
optimizer state; to evaluate those weights directly, pass state.averaged as
variables. step keys the evaluation’s RNG and labels the result.
schedule_step is the step the objective’s schedules read, and defaults to
step. When training attempts were rejected, pass the number of accepted
microbatches (TrainState.microstep, as fit does) so the schedule follows
only the work training accepted. updates is the updates training
committed (Step.updates), and defaults to the schedule’s step.
warm scores at most WARM_BATCHES batches and finalizes no score, so
the evaluation holds none: a pass over them runs every program a longer
pass does, the loss’s reduction included, which a run compiles this way
before its first step (Trainer.fit), and no score of a part of the
split is checked or read.
scored_to, a directory, takes every batch’s scoring artifacts, process 0 writing each batch’s
real rows (a decision run’s Decisions) as <scored_to>/<split>-<batch>.npz
(dew.training.tracker.write_artifact), so a pass’s metrics pool exactly offline, across
batches and splits; a warming pass writes none. Process 0’s scored_to is the pool’s: every
process scores the batches it writes, whatever it was given.
The scores are broadcast to every process. Previews stay on process 0, and there is at most one objective preview, however long the validation split is. Reporting the result is up to the caller. In a process pool, the callers must agree on whether reporting failed before they enter their next collective, and a device failure in flight still requires ending the distributed runtime.
Inspection
Section titled “Inspection”class Inspection( paths: tuple[str, ...] = (), targets: tuple[Target, ...] = ('params', 'grads', 'updates', 'deltas'), every: int = 100, modules: tuple[str, ...] = (), cotangents: bool = False, histogram: bool = False,)What a run watches and how often (row 2317).
paths are globs over a parameter’s path under params (dew.optim.parameter_name:
layers_0/attn/q_proj/kernel, *norm*), as a ParamGroup’s patterns are; each leaf a glob
selects is summarized on its own, for each target: its mean, standard deviation, root mean square,
largest magnitude, non-finite share and zero fraction. Every sampled step also reports each
target’s global L2 norm, over every parameter, and the update-to-weight ratio,
||update|| / ||params||. every is the windows of the run’s accumulation steps between samples
(samples): the update that commits in each, every every-th update of a run no step of which is
rejected; the run’s logs carry the latest sample, so a cadence of the logs’ or a multiple of it reports
each sample once.
modules: tuple[str, ...]-
Modules whose outputs a sampled step summarizes, by a glob of their class name (
*Attention,MLP) or their path as the log names it (layers_3/*, a layer by its place in the decoder whether the stack scans or pipelines it): each call’s output, its first array where it returns several, over a language model’s real tokens (dew.nn.inspection.watched_positions), one summary a layer in a scanned stack and one a microbatch in a pipelined one (dew.nn.inspection.watching), logged asinspect/activations/<module path>/<statistic>, a pipelined layer by its place in the stack. cotangents: bool-
Whether a sampled step also summarizes the loss’s cotangent at each watched output, over the same positions: its mean, standard deviation, root mean square, non-finite and zero shares (and histogram), from sufficient statistics a probe’s gradient carries (
dew.nn.inspection.tapped), so no tensor of the output’s size is kept, logged asinspect/cotangents/<module path>/<statistic>. The sampled microbatch’s own, its pipeline’s microbatches summed; no parameter or update changes with it. histogram: bool-
Whether each summary also holds the share of its magnitudes in each decade (
dew.nn.inspection.HISTOGRAM_EDGES), logged one scalar a bin. watched: Watched | None-
The activations a sampled step watches, where
modulesnames any.
Inspection.samples
Section titled “Inspection.samples”def samples(step: int, accumulation: int) -> boolWhether the step that ends at step is sampled: each step of every every-th window of
accumulation steps, the loop’s count of the steps it attempts, so the update that commits in it
is sampled at the window’s end or early, where a run on the examples clock closes its last window
short. A rejected step (a non-finite loss under a dynamic loss scale) drops its window and shifts
the commits after it, which the loop sees only on the devices: a sampled window that then holds
no commit skips its sample, as often as rejections so fall.
LocalTracker
Section titled “LocalTracker”class LocalTracker(directory: str | Path, *, plots: bool = False)Writes each report to a tracking directory as it is logged.
The reports are JSONL journals (scalars.jsonl and records.jsonl),
preview files and, with plots, plots. Nonfinite metrics are written to
the journal as the strings NaN, +Inf and -Inf. Plots are drawn only by
plot(), or by close when plots is set, as dew.figures curves, and
they read the journal, so the tracker keeps no history in memory.
LocalTracker.log
Section titled “LocalTracker.log”def log(scalars: Mapping[str, float], step: int) -> NoneLocalTracker.artifact
Section titled “LocalTracker.artifact”def artifact(value: Reported, step: int) -> NoneLocalTracker.scalars
Section titled “LocalTracker.scalars”def scalars() -> list[tuple[int, dict[str, float]]]The scalars the journal holds, each report’s step and values in the order they were
logged, nonfinite values read back as floats. A tracker over a run’s directory reads the
run’s reports this way, its final validation’s scores among them (Trainer.fit logs them
under the run’s last step); a journal no report wrote yet holds none.
LocalTracker.timed
Section titled “LocalTracker.timed”def timed() -> list[tuple[int, float, dict[str, float]]]scalars, each report with the wall-clock second it was logged at (time.time()) between
its step and its values, which a run’s reports against the time it took read as a curve.
LocalTracker.plot
Section titled “LocalTracker.plot”def plot( *metrics: str = (), smooth: int = 1, y_label: str | None = None, y_range: tuple[float, float] | None = None,) -> FigureThe journal’s metrics as one figure (dew.figures.curve), each line at the steps its
metric was logged: by default the training loss beside every split’s validation loss (each
<prefix>/loss the journal holds). smooth draws each line’s running mean over that many
points over it, and y_label names the axis, by default the metrics’ common last part
(loss). A nonfinite value is left out of its line and counted on the figure; the journal
keeps it. Raises KeyError for a metric the journal never logged, naming those it holds.
LocalTracker.plot_files
Section titled “LocalTracker.plot_files”def plot_files() -> list[Path]Draw each metric in the journal to its own SVG file and return the paths. A file is named
for its metric, each character outside [A-Za-z0-9_.-] written -, with a digest of the
whole name, so two names never share a file. The journal is not changed.
LocalTracker.close
Section titled “LocalTracker.close”def close() -> NoneMLflowTracker
Section titled “MLflowTracker”class MLflowTracker(experiment: str, name: str | None = None, *, uri: str | None = None)An MLflow run in experiment, opened on the first value logged into it.
Scalars are logged as the run’s metrics, and a preview is uploaded as the
files LocalTracker writes for it. A record is uploaded as a JSON
artifact and not as a parameter, because MLflow refuses a second value
for a parameter and a run reports several records of the same type.
run: tuple[MlflowClient, str]-
The MLflow client and the run id; the first access creates the run, and the experiment if it does not exist.
MLflowTracker.log
Section titled “MLflowTracker.log”def log(scalars: Mapping[str, float], step: int) -> NoneMLflowTracker.artifact
Section titled “MLflowTracker.artifact”def artifact(value: Reported, step: int) -> NoneMLflowTracker.close
Section titled “MLflowTracker.close”def close() -> NoneObserve
Section titled “Observe”class Observe( every: int, call: Callable[[TrainState], Any], values: list[tuple[int, Any]] = list(),)call(state) at every every-th step of a fit and at its last, its result kept in values.
The state is lent for the call alone: what call returns is copied to the host, so a returned
tree of the state’s own arrays (lambda state: state.averaged) is a snapshot, not a view of
buffers the next step reuses. A failure in call on any process fails the fit on all of them,
at that step. values holds (step, value) in step order; a fit that resumes from a checkpoint
drops what it holds past the restored step, which a failed run observed and the resumed one
retrains, and a fit that finds the run already complete observes its final state once.
Observe.opened
Section titled “Observe.opened”def opened(step: int, state: TrainState, *, complete: bool) -> NoneA fit begins at step, restored or fresh: agree the cadence and forget later values. A
fit with nothing to train, its run complete at step, observes the state there where any
process holds no value for it, every process alike.
Observe.at
Section titled “Observe.at”def at(step: int, state: TrainState) -> NoneAfter step, where the cadence asks for it.
Observe.last
Section titled “Observe.last”def last(step: int, state: TrainState) -> NoneThe run ended at step, done or stopped by a preemption: observe it unless this fit
just did.
Observe.take
Section titled “Observe.take”def take(step: int, state: TrainState) -> Nonecall(state) on every process, its result copied home and kept as step’s value. A fit’s
steps only rise past what opened kept, so the value is the ledger’s new tail or replaces
it.
PassesProgress
Section titled “PassesProgress”class PassesProgress( make: Callable[[Objective[L, E], optax.GradientTransformation], Callable[[TrainState, Batch], tuple[TrainState, jax.Array, Aux]]],)A custom training step (Trainer(step=)) that passes the run’s progress to a model that reads
it (dew.nn.progress, Progress(state.updates)), as the objective’s own loss does: make, its
factory, wrapped. A custom step not so declared refuses such a model, which it would otherwise
train at the run’s end.
ProfileWindow
Section titled “ProfileWindow”class ProfileWindow( directory: str, steps: int, warmup: int = 2, capture: telemetry_profile.CaptureOptions = telemetry_profile.CaptureOptions(), reports: bool = True, wait: bool = False,)A request for one profiler window per fit.
It traces steps steps into directory after warmup steps have run and the
run’s first step has compiled, so the trace shows the loop and not the compile, and it holds the
training steps and the loop’s own cadenced reads of them (its log, an observer’s call): the work between
them that is no step’s (an evaluation, a checkpoint save, a cooldown branch) runs outside its capture,
which starts again at the next step’s read, so a window that spans such work writes a capture of each
run of its steps (dew.training._fit.FitRun.outside_window). A fit under JAX’s AutoPGLE, which
profiles programs in sessions of its own, refuses a window (PGLE_CAPTURE).
The capture records what capture asks (dew.telemetry.profile.CaptureOptions): by default
host events at level 2, no Python calls (the Python tracer slows Python-heavy host work several
times over), and on GPU at most its activity_cap CUDA activity events.
A window that reports nothing keeps its trace and manifest alone, which needs no XProf and its stop
writes at once; one that reports converts its capture’s reports in a process of its own while the run
trains on, waited for at the fit’s end, minutes for a training window, or, where it waits, in the
run’s own process as its capture stops, before the run’s next step, as Profiler.stop(wait=True)
does: a short interactive run’s, or one whose sandbox forbids it to create a process (a live page’s
kernel). A run’s own window (RunProfile, every run’s by default) is
resolved into one by its config (dew.config.TrainerConfig.window), kept with its records.
dew.Profiler is the other way to capture a trace, as a context manager
around any code; a fit refuses to schedule a window inside one. The loop
reports the window it wrote as the ProfileTraced record of
dew.telemetry.records.
Rollout
Section titled “Rollout”class Rollout(Protocol)Produces a batch on the host, before the compiled step reads it.
Sampling has side effects and cannot be traced, so it runs outside jit. The
trainer calls the rollout with the state, the prefetched batch and a key
folded from the run key and the step, then reshards the result with
shard_batch. The returned batch must hold arrays of fixed shapes, so the
step still compiles once per run.
A rollout may have metrics, a mapping of names to floats about its latest
call (rewards, lag, truncation); each logging interval sends them to the
tracker and the display as rollout/<name>. It may declare how the display
shows them in shown, as an objective does.
A rollout whose draws_further is true may read data batches past the
step’s, as DAPO’s dynamic sampling does until its training batch is full:
the trainer calls it with further, a read of the training stream’s next
batch, None where it ran out (FitRun.further), every process reading
as many, whose position the step’s checkpoint saves as any read’s.
A rollout whose calls change more than the batch they return (its metrics,
a log, a journal, rollouts in flight) may have branch(name), a context
manager a cooldown branch’s calls run inside, on the branch’s states, name
the branch’s own (cooldown-<step>, the step it branches at), after which
the rollout stands as the run left it (_branched). One without it is
called alike there.
A rollout whose next batches depend on state it carries between calls (its
environments mid episode, a collection it serves in minibatches, a replay)
may have saved(), that state as a mapping of arrays (host arrays of 32
bits or fewer a leaf, which the trainer places), None before its first
call; saved_shapes(), its leaves’ shapes and dtypes, readable before any
call; restored(tree), which carries on from a saved tree; and
branch(name), so a cooldown branch leaves that state as the run left it.
Each checkpoint of the run holds the run’s rollout’s tree beside its state
(dew.training._fit._rollout_saved), and a resume hands it back, placed
as the run’s key is (_rollout_resumed), so a resumed run samples the
batches the run that never stopped did. It is the pool’s: a run of several
processes with a checkpointer is refused one as its fit starts
(_rollout_admitted). The
contract is the post-training spec’s (docs/design/post-training.md, §14).
RunProfile
Section titled “RunProfile”class RunProfile( steps: int = 2, warmup: int = 10, capture: telemetry_profile.CaptureOptions = telemetry_profile.CaptureOptions(),)A run’s own profile window, every run’s by default (dew.config.TrainerConfig.run_profile): steps
traced once warmup have run past its first compile, kept with the run’s records, its trace and
manifest alone (no reports: no XProf asked, no conversion the run’s end waits for), and taken where the
process can take one (capturable), none where it cannot, where a ProfileWindow a run names refuses.
Its config resolves it into its window (dew.config.TrainerConfig.window); a trainer is given that.
RunProfile.window
Section titled “RunProfile.window”def window(directory: str) -> ProfileWindowThis window as a trainer traces it, into directory.
TensorBoardTracker
Section titled “TensorBoardTracker”class TensorBoardTracker(directory: str | Path)A TensorBoard event file in directory, opened on the first value
logged into it.
Scalars are written as scalar summaries. A record is written as the JSON
LocalTracker journals for it, in a text summary under
reporting/<type>. Image and video grids are written as image summaries,
with a video as an animated GIF; text samples as text; and
representations as a histogram of their spread. Other previews raise
TypeError.
writer: EventFileWriter-
The event-file writer, which the first access opens in
directory.
TensorBoardTracker.log
Section titled “TensorBoardTracker.log”def log(scalars: Mapping[str, float], step: int) -> NoneTensorBoardTracker.artifact
Section titled “TensorBoardTracker.artifact”def artifact(value: Reported, step: int) -> NoneTensorBoardTracker.close
Section titled “TensorBoardTracker.close”def close() -> NoneTracker
Section titled “Tracker”class Tracker(Protocol)Tracker.log
Section titled “Tracker.log”def log(scalars: Mapping[str, float], step: int) -> NoneTracker.artifact
Section titled “Tracker.artifact”def artifact(value: Reported, step: int) -> NoneTracker.close
Section titled “Tracker.close”def close() -> NoneTrackers
Section titled “Trackers”class Trackers(*trackers: Tracker = ())Sends each report to several trackers.
Every tracker receives every report, even when an earlier tracker fails. Then the first failure is raised, with the later ones attached as notes.
Trackers.log
Section titled “Trackers.log”def log(scalars: Mapping[str, float], step: int) -> NoneTrackers.artifact
Section titled “Trackers.artifact”def artifact(value: Reported, step: int) -> NoneTrackers.close
Section titled “Trackers.close”def close() -> NoneTrainer
Section titled “Trainer”class Trainer( objective: Objective[Loss, Effects], optimizer: optax.GradientTransformation | OptimConfig, *, key: int | jax.Array, mesh: MeshSpec | Mesh = _DEFAULT_MESH, layout: Layout = _DEFAULT_LAYOUT, microbatches: int | None = None, accumulation: int = 1, split: int | None = None, dynamic_scale: bool | dynamic_scale_lib.DynamicScale = False, checkpoints: Checkpoints | None = None, tracker: Tracker | None = None, step: Callable[[Objective[Loss, Effects], optax.GradientTransformation], StepFn] | None = None, rollout: Rollout | None = None, profile: ProfileWindow | None = None, lower_for: str | None = None, ahead: AheadOfTime | None = None, tune_kernels: bool = False, record: bool = False, links: Mapping[str, Link] | None = None, inspection: Inspection | None = None, param_dtype: str | None = None, diloco: DiLoCo | None = None,)Runs an Objective: gradients, sharding, EMA, checkpoints and logging.
key: jax.Array-
The run’s PRNG key: the one it was given, or its seed’s, made when first read, which every path that builds state does, after readying the process (
ready_process) and outside any trace: constructing a trainer opens no backend before its flags and cache are set. device_mesh: Mesh-
The device mesh (
device_mesh), built on first use. microbatches: int-
The microbatches a pipelined step feeds through the stage axis (
stage_microbatches).
Trainer.initial_state
Section titled “Trainer.initial_state”def initial_state( initializer: Initializer | None = None, key: int | jax.Array | None = None,) -> TrainStateBuild the state a fresh run starts from, a DiLoCo run’s with its groups’ copies (DiLoCo.started),
every layer whole, which a stage-local step stacks by stage (_staged_start). It is pure, so fit
traces it once for its shapes and once, sharded, for its values.
Both inputs default to the run’s own, and place passes them explicitly so that what it compiles
takes them as arguments. A held checkpoint then reaches the device as an argument instead of a
constant embedded in the executable. Passing None means “use the configured input”, which is what a
call without arguments does. This is the only implementation of the state, so a subclass overrides it
here and every path sees the override.
Trainer.staged
Section titled “Trainer.staged”def staged(state: TrainState) -> TrainStatestate (initial_state’s, stored’s) held and placed as the step holds it, as fit starts.
Trainer.stored
Section titled “Trainer.stored”def stored(state: TrainState) -> TrainStatestate with a stage-local run’s layers back on layers_N, whole on every stage, every mirror
alike: for a reader by layer name (an export), at the cost of every layer on every device.
Trainer.reading
Section titled “Trainer.reading”def reading( loader: Loader, prefetch: int = PREFETCH_DEPTH, *, examples=None, steps=None,) -> NoneTake loader’s device steps, prefetch and clock (clocked), as fit and compile-ahead do.
Trainer.shardings
Section titled “Trainer.shardings”def shardings(state: TrainState) -> Placement[TrainState]Return where each field of state is placed, on the axes that suit its kind.
Parameter gradients follow parameters, replay records follow batches, and the layout’s host-resident
fields go in pinned host memory. Under a CPU-owned state, the frozen collection is the exception: it
stays where the realization reads it (execution.resident) for the whole run.
Trainer.validate
Section titled “Trainer.validate”def validate( batch: Batch | None = None, *, steps: int | None = None, capacity: int | None = None,) -> ValidationWhat fit checks of the state before it places it (Layout.validate), read without
placing anything, so a run is checked on a CPU host before a costly launch.
The state is a fresh run’s: its parameters, optimizer state and EMA, and, given one global batch as
the run reads it, the accumulation its steps pool where accumulation is above 1, read in the first
of the splits its plan may take whose devices hold it, the trainer’s own where it fixes one
(splits). It leaves out the narrow copies a step may read (TrainState.compute), which the plan
drops first where the step does not fit with them. An OptimConfig is built over steps, as
fit(steps=...) builds it. capacity is the bytes each device has, for a mesh whose devices report
none: a CPU mesh of --xla_force_host_platform_device_count devices standing in for GPUs.
Raises what fit raises before placing the state: LayoutRefused for a state over a device’s memory
or a layout that replicates too much of the model, and ValueError for rules that name an axis the
mesh lacks.
Trainer.place
Section titled “Trainer.place”def place() -> tuple[TrainState, Placement[TrainState], bytes | None]Put the state on the mesh, fresh or restored, once validate’s checks pass.
It returns the state with its shardings and the data position a resume continues from.
Trainer.compile
Section titled “Trainer.compile”def compile( state: TrainState, batch: Batch, trials: Trials | None = None, *, capacity: int | None = None,) -> CompiledStepCompile a transaction over the state and one already-produced global batch.
The step consumes its state: new = step(old, batch) donates the old buffers, keeping one
copy of parameters and optimizer state. Keep no reference to the old state after stepping.
Orbax copies a checkpoint to host before save returns, provided Checkpoints names neither
prioritized keys nor a concurrent transfer limit. The loader’s batch is not donated.
The plan (dew.training.planning) takes the fewest microbatches at least recomputation, a
transaction pooling split x accumulation. Batches split as the first plan or the pooling
state’s window holds; later shapes climb the ladder alone. No admitted plan compiles or times
optional alternatives, even with trials: speed selection is offline. Where JAX’s AutoPGLE
is on (jax_enable_pgle), the plan admitted runs its first runs on a trial, trials’ or, where
none are given, a copy of the state beside it, refused where the devices hold no room for it
(Trials.profiled), and its program compiles again guided by their profile (Planned.guided),
the executable it replaces released with its program (Planned.released). A device that states
no memory, a compile-only GPU’s, holds capacity bytes, as a step compiled ahead states them.
Trainer.fit
Section titled “Trainer.fit”def fit( dataset: Dataset | Loader | Handed, *, steps: int | None = None, epochs: int | None = None, tokens: int | None = None, examples: int | None = None, log_every: int | str | Counted = 100, eval_every: int | str | Counted | None = None, checkpoint_every: int | str | Counted | datetime.timedelta | None = None, checkpoint_final: bool = True, metrics: Sequence[Metric] = (), preview: bool = False, scored_to: str | os.PathLike[str] | None = None, best: str | Metric | TrainingScalar | types.MethodType | Best | Sequence[Best] | None = None, stop: Plateau | Pause | None = None, validation: HandedSplits | None = None, summaries: Sequence[Summary] = (), restore_best: bool = False, state: TrainState | None = None, observe: Observe | None = None, device_prefetch: int = PREFETCH_DEPTH, resume: Resume = 'auto', cooldowns: Cooldowns | None = None,) -> TrainStateTrain until the step or real-token cap is reached, resuming the latest checkpoint as resume says;
return the state as the run holds it, which its model reads off the mesh (off_stages).
resume="auto" continues the checkpointer’s latest checkpoint where it holds one and otherwise
starts afresh; “require” refuses to start afresh, for a job that must continue, and “never” refuses
a directory that already holds a run (Checkpoints.resumed). An explicit state is the run’s
initial state, in place of initializing one and of restoring a checkpoint (a float64 truth’s widened
copy of another run’s, say), not a continuation, which is a checkpoint’s alone: its input reader
starts at the position dataset supplies.
Every log_every steps, the tracker receives the loss, the objective’s metrics and the throughput.
tokens stops at the first microbatch that reaches this real-token budget, the tokens committed
updates trained on (train/committed_tokens), persisted in checkpoints. Padding never counts.
With both caps the first reached ends the run; with steps=None, a decaying schedule needs its own
step horizon. examples trains on the chain’s first source examples, each once with all its rows
(Loader.example_steps).
Every eval_every steps, and at the end, the validation split is scored. The objective’s
artifacts go to the tracker and to metrics, whose reductions are logged as val/<name>.
validation is that split as a Dataset, or names several, each a Dataset, a reader scored as
dataset.val would be or an EvalSuite with its own metrics and cadence, logged as
<name>/<metric>. Each of summaries weighs those scores into one, evaluation/summary/<name>,
formed at a step that scored every split it weighs, which best and Plateau rank by.
restore_best=True returns the model the run ranked best: the final state with the counters,
variables and EMA of the best step (MODEL_STATE), read from its best checkpoint, a weights-only
one included, or without Checkpoints from the copy it held (HeldBest). Its optimizer state
and the rest stay the final step’s, so it is the best model, not a point to resume training
from, which the run’s checkpoints are. cooldowns branch the run into cooldowns, each scored by
every suite as cooldown/<split>/<metric> and discarded (dew.training.cooldown.Cooldowns).
Every checkpoint_every steps (or “5%”, FitPlan.cadences), the state and data position are written;
checkpoint_final=True also writes the final state, independently of that cadence. A duration is
checked at each log_every step, so they are written at the first log step after that much time, and
every checkpoints.local_every steps to the local directory. A preemption notice (a scheduler’s
SIGTERM; PreemptionNotice) stops the run at the next step every process agrees on. That step’s state
and data position are written, the final validation is skipped, and fit raises Preempted, which ends
the program with status 143 unless caught. Run it again and fit resumes from there.
Previews are generated only when preview=True and a tracker receives them; scalar reporting never
triggers preview work. scored_to, a directory, takes every validation batch’s scoring artifacts at
each evaluation, <scored_to>/step-<step>/<split>-<batch>.npz (Evaluation.run), a cooldown
branch’s under cooldown/ there. observe takes values of the state at a cadence of its own while
the run trains (Observe). device_prefetch training batches are placed on the devices ahead of
the step, which the step’s memory plan holds room for. The process is readied first
(dew.training.runtime.ready_process), as prepare_process readies it: a program that never called
that gets Dew’s log handler, and one whose JAX opened on the CPU beside an NVIDIA GPU stops.
Trainer.scheduled
Section titled “Trainer.scheduled”def scheduled(updates: int | None) -> NoneBuild the configured optimizer (an OptimConfig) for updates updates, as fit and
validate do for their run’s length; its state holds the rate it logs. A trainer given an Optax
transformation keeps it.
class Wandb( project: str, entity: str | None = None, offline: bool = False, name: str | None = None, shared: bool = False,)Where a run reports to Weights & Biases.
Setting it turns tracking on; the entity and the offline switch have no effect
without a project. name is the run’s display name there, apart from the name
its checkpoints are kept under; None shows it as the run is named. shared
opens the run in W&B’s shared mode as its primary writer, so other processes
(a forecaster, an evaluator) log into the same run on axes of their own
(WandbTracker’s primary=False), whose rows a run opened plainly drops: two
processes on one run, 30 rows and 0 plainly, 30 and 3 shared (wandb 0.31.0).
A shared run is one W&B’s server holds, so an offline one is refused (WandbTracker).
WandbTracker
Section titled “WandbTracker”class WandbTracker( project: str, name: str | None = None, *, entity: str | None = None, config: Mapping[str, object] | None = None, offline: bool = False, id: str | None = None, shared: bool = False, primary: bool = True, label: str = 'train',)A Weights & Biases run, opened on the first value logged into it.
shared opens it in W&B’s shared mode (wandb.Settings(mode="shared"),
wandb 0.19.9 or later and W&B Server 0.68 or later), where several processes write one run, each its
own label: the primary writer opens it and logs on train/step; any
other attaches to the run’s id, logs its label’s metrics on its own
step (<label>/step) and leaves the run’s state to the primary
(x_update_finish_state), as W&B’s distributed logging guide has it.
WandbTracker.log
Section titled “WandbTracker.log”def log(scalars: Mapping[str, float], step: int) -> NoneWandbTracker.artifact
Section titled “WandbTracker.artifact”def artifact(value: Reported, step: int) -> NoneWandbTracker.close
Section titled “WandbTracker.close”def close() -> Noneinspected
Section titled “inspected”def inspected( trainer: Trainer[LossT, EffectsT], batch: Batch, *, steps: int | None = None,) -> dict[str, float]The scalars trainer’s inspection logs, of one sampled update over batch from the state the
trainer places, its checkpoints’ latest where it has them, else its start: the view a live run
logs, for a loaded run on one batch offline. A trainer that accumulates steps to the end of its window,
from where its state stands in it (a fresh window or a restored part of one), batch each of its
microbatches, and reports the step that commits. An OptimConfig is built over
steps, the run’s length, as fit(steps=...) builds it. The stepped state is discarded.