Skip to content

dew

Dew: one objective, one trainer.

Each name exported here is imported from its own module when you first access it, not at import dew. So import dew.training loads only the training layer, with no modality, encoder or tracker backend. Importing dew opens no JAX backend and loads no optional dependency; encoders, decoders and datasets load what they need when they are built.

import dew does set XLA flags, before anything it imports loads jaxlib, so they hold for every XLA runtime the process loads (dew._devices.flags_hold): one everywhere and, when JAX’s CUDA plugin is installed, two more. --xla_allow_excess_precision=false makes XLA round values declared in a narrow dtype such as bf16 where the program rounds them. On CUDA, --xla_gpu_enable_allocator_spatial_partitioning=false keeps a preallocated GPU pool in one piece for a step’s temporaries, and --xla_backend_extra_options=-nvptx-prec-divf32=2 divides fp32 to the nearest float, as a CPU and the references Dew is held to do (dew._devices). If you set any of them yourself in XLA_FLAGS, your value is kept. Where jax was imported before Dew, XLA may have parsed its flags already and Dew’s are unproven; where the JAX backend has already opened, they cannot take effect, and Dew logs a warning.

import dew also makes the JAX user contexts the kernel autotuner and kernel choices key traces on (dew._cache.KERNEL_TUNES, KERNEL_PINS, KERNEL_CHOICES): JAX makes one only beside no other JAX call, so before any thread starts.

import dew leaves the process’s loggers alone and asks nothing of its hardware. Dew’s log handler, and the refusal of a machine with an NVIDIA GPU whose JAX opened on the CPU, come with the process’s setup: prepare_process, or the first Trainer.fit of a program that never called it (dew.training.runtime.ready_process).

import dew marks absl’s flags parsed when nothing has parsed them, as absl.app would: Grain’s worker processes read absl flags as they start, and a program that never ran absl.app would stop them there. A program that parses its flags itself, through absl.app or FLAGS(argv), still does.

NameSummary
CFGInterval-limited classifier-free guidance. Documented in dew.sampling.guidance.
AuxEverything a loss returns besides its statistics. Documented in dew.objectives.base.
BestKeeps the top best evaluations by a metric, a recorded metric name, or a custom score. Documented in dew.training.
CheckpointsManages the checkpoints of one run in one directory.
ConditionNames one conditioning input: its encoder, the batch field holding its tokens, and the raw datum for the unconditional branch. Documented in dew.inputs.
DatasetRecords a run reads: a store (Files, Hub, TFDS, Memory, Torch), or a store with the steps the verbs added, in the order they are written. Documented in dew.data.pipeline.
EMASpecWhich leaves the EMA copy tracks, and how fast it follows them. Documented in dew.objectives.base.
EvalSuiteA validation split fit scores with its own metrics on its own cadence. Documented in dew.training.
EvaluationThe result of one evaluation, with its previews kept on process 0. Documented in dew.training.
FieldA batch field and its per-example shape, such as Field("image", (128, 128, 3)). Documented in dew.inputs.
ImageGridImages in [-1, 1], [N, H, W, C], with the text each was conditioned on where there was any. Documented in dew.artifacts.
InputSpecNames the sample field and the conditions, keyed by the model keyword each is passed under. Documented in dew.inputs.
KeepKeeps the union of the latest steps, periodic steps, wall-time-spaced steps and a predicate’s steps.
LayoutDecides how a train state is placed on a mesh. Documented in dew.distributed.placement.
LocalTrackerWrites each report to a tracking directory as it is logged. Documented in dew.training.
MLflowTrackerAn MLflow run in experiment, opened on the first value logged into it. Documented in dew.training.
MeanAverages a metric over a validation pass, from per-example values or additive (total, count) pairs. Documented in dew.eval.
MeshSpecDevice counts for the mesh’s sharding axes; the data axis takes the devices left over. Documented in dew.distributed.placement.
ObjectiveDefines what is learned: the parameters, the loss and what evaluation produces. Documented in dew.objectives.base.
Observecall(state) at every every-th step of a fit and at its last, its result kept in values. Documented in dew.training.
PauseEnds the run once it has trained fraction of its budget, its state and data position written in full, so a run resumed from that checkpoint trains on as one that never paused: the budget still sizes the learning-rate schedule. Documented in dew.training.budget.
PlateauStops training after evals eligible evaluations without an improvement larger than min_delta. Documented in dew.training.budget.
ProcessPairs a noise schedule with what the model predicts on it and how the loss is weighted. Documented in dew.diffusion.process.
ProfileWindowA request for one profiler window per fit. Documented in dew.training.
ProfilerCaptures a JAX profile of this process, as a context manager or through start and stop. Documented in dew.telemetry.profile.
RepresentationsEncoder outputs [N, D] and the labels of the records they came from, for a probe to score. Documented in dew.artifacts.
StepWhat the trainer tells an objective about the current step. Documented in dew.objectives.base.
SupervisedTrains model, a Flax module or a loaded source whose weights training starts from, on loss, reporting metrics beside it. Documented in dew.objectives.supervised.
TensorBoardTrackerA TensorBoard event file in directory, opened on the first value logged into it. Documented in dew.training.
TextSamplesGenerated token rows, with optional decoded preview text and prompt. Documented in dew.artifacts.
TokenScoresTeacher-forced per-token losses [N, L] and the weight of each target. Documented in dew.artifacts.
TrackerDocumented in dew.training.
TrackersSends each report to several trackers. Documented in dew.training.
TrainStateHolds everything a run must checkpoint to resume where it stopped. Documented in dew.state.
TrainerRuns an Objective: gradients, sharding, EMA, checkpoints and logging. Documented in dew.training.
VideoGridClips in [-1, 1], [N, T, H, W, C], with the text each was conditioned on where there was any. Documented in dew.artifacts.
WandbTrackerA Weights & Biases run, opened on the first value logged into it. Documented in dew.training.
pipelineLoad the inference task for source, with its weights placed once. Documented in dew.inference.
sampleRun the reverse process from x_T with solver and return the final sample. Documented in dew.sampling.

class source

class Checkpoints(
directory: str,
*,
keep: int | Keep = Keep.latest,
local_directory: str | None = None,
local_every: int | str | None = None,
)

Manages the checkpoints of one run in one directory.

Constructing one opens nothing; the orbax managers are created on first use. The directory keeps the latest keep steps, so a resume has something recent, plus the step with the lowest loss metric a save reported. A save without metrics can never become the best step.

local_directory names a path on every host’s own disk where the run keeps one more checkpoint, the latest, which fit writes every local_every steps, or at each share of the run it names, such as 5%, as fit resolves its other cadences (dew.training.budget.every_step). Each process writes the shards its devices hold under its own subdirectory, so the same path works for a pod and for one host running several processes. latest is the newest step every process can read, local or persistent, and restore reads it from wherever it is. A local checkpoint restores onto the placement it was written with, because no process holds another process’s shards; the persistent checkpoint restores onto any mesh.

local_path: str

Return this process’s own local directory.

latest: int | None

Return the newest committed step a resume can read, local or persistent (_newest).

def would_keep(ranking: Ranking | Sequence[Ranking]) -> bool

Return whether an evaluation would enter the best-K set of any of its trackers.

def committed() -> list[int]

The steps whose checkpoints have committed, oldest first, read from the store at each call without opening it or writing to it, as latest reads it: what a process beside the one that trains (an evaluation worker) reads as the run saves, a save in flight listed once it commits. kept is this process’s own view, its retained steps and their metadata.

def kept() -> list[Kept]

Return the committed retained steps, oldest first; their immutable metadata is cached. The steps are those this process’s store saw open or saved itself; a process that reads a run another saves into lists its new steps with committed.

def python_run(reasons: Sequence[str]) -> None

Why no record rebuilds the run these checkpoints keep, each part of it a run’s record cannot state, which PYTHON_RUN carries where a run defined in Python first saves (Trainer.fit names them from its own fields) and RunConfig.read shows.

def cooldown(step: int) -> Mapping[str, object] | None

The cooldown branch the run began at step and has not finished, as it was resolved there (dew.training.cooldown); None where none is (COOLDOWN_FILE).

def cooling(pending: Mapping[str, object] | None) -> None

Keep pending as the cooldown branch the run has begun, or with None that it has none unfinished; process zero writes it, beside the steps (cooldown).

def artifact(step: int | str | None = None) -> JSON

Return the selected step’s inference declaration, from its custom metadata alone (_custom), as control reads a committed step’s: the manager’s metadata opens every array’s TensorStore besides (1.7 s of a U-Net run’s reload, which reads it before anything, row 3763); a local step’s as its process holds it.

It is None when the step’s objective declares no inference record.

def control(step: int) -> dict
def pinned(step: int | str | None = None) -> int

The exact step step selects: itself, the best by a ranking, or, for None or latest, the newest committed now that holds a model (a step kept for its model alone among them). A later save does not move it.

def resolve(step: int | str | None) -> int | None

The step step selects (pinned), None where it is None and no step holds a model.

def resumed(resume: Resume = 'auto') -> int | None

The step a run continues under resume, the latest, or None where it starts afresh: “auto” continues one where the directory holds it; “require” refuses a directory with none, so a job that must continue never trains from scratch; “never” refuses one that holds a run, which a run started afresh would write beside.

def path(step: int) -> str
def source(step: int) -> str

Return the directory restore reads step from.

That is this process’s local directory when the step is the local one every process holds, and the persistent directory otherwise; a local step this process lost is read from the first process’s copy that holds it.

def save(
step: int,
state: TrainState | Mapping[str, object],
saved: bytes | None = None,
metrics: Mapping[str, float] | None = None,
*,
share: DataPartition | None = None,
ranking: Ranking | Sequence[Ranking] | None = None,
control: dict | None = None,
weights_only: bool = False,
primary: str | None = None,
rung: JSON = None,
artifact: JSON = None,
boxed: Mapping[tuple[str, ...], LogicalAxes] | None = None,
rollout: Mapping[str, object] | None = None,
) -> None

Write state under step, asynchronously, a train state with its rollout’s tree of arrays where it holds one (rollout), which rollout_state reads back.

state is a run’s TrainState, or a mapping of arrays in its place for a state that is not a training run’s, such as a simulation’s, which has no optimizer, average or loss scale. boxed is the axes the model’s modules box on their own parameters (dew.nn.sharding.boxed_axes), which the state holds unboxed; variables places with them. A mapping is written as it is, with metrics, ranking and control as a train state’s are, and restore reads it back through a mapping template; it takes no data position, share, weights-only split, rung or artifact. It is written whole at every step, without the store a train state’s static collections go to (FROZEN_STORE), so large arrays no step changes are better kept out of it.

Sharded arrays go straight to Orbax. Gathering them onto the host first would serialise the whole state through one process and defeat the purpose of an async checkpointer. A stream reports its position as JSON bytes, which tensorstore has no dtype for, so the bytes are stored as uint8 rows, one per process, next to the data share that process read. A global position and a share’s offset are stored the same way and told apart on restore. A position without its share is refused, since no reader could be matched to it. A failed write surfaces from wait, which deliberately does not catch it: a checkpoint that did not land is lost data.

A state with arrays in pinned host memory is written before this returns, because the write reads those arrays’ own buffers and the next step donates them. Giving Orbax a copy instead would make the next step wait only for the copy, but it would hold a second copy of that state in pinned host memory until the write lands: 12 bytes a parameter for fp32 Adam moments and EMA, 84 GB at 7B parameters, on hosts that keep the state there because device memory is short.

def profile_steps() -> list[int]

Return the complete checkpoints that hold post-hoc EMA snapshots, oldest first.

def profile_metadata(step: int) -> tuple[int, tuple[float, ...]]

Return a snapshot’s recorded updates and relative standard deviations, without its averages: the run’s updates, under DiLoCo the first group’s, whose clocks are the run’s (_profile_updates).

def restore_profiles(step: int) -> tuple[Variables, ...]

Read only a snapshot’s averages, as host arrays, from its retained checkpoint: under DiLoCo each group’s, stacked over the groups on a first axis (_profiles_at).

def posthoc_ema(std: float, step: int | None = None) -> Variables

Return the post-hoc EMA with relative standard deviation std at checkpoint step.

step defaults to the latest snapshot. The result is host arrays in the params’ structure and dtypes. It sums every snapshot up to step, from every tracked profile, with the weights coefficients solves for, reading one snapshot at a time and accumulating in fp32 or wider. Use the result where the run’s params go: merge(variables, {"params": checkpoints.posthoc_ema(...)}).

Under DiLoCo the profiles are each group’s, of the parameters its inner solver makes (DiLoCo.started moves the configured optimizer into the groups), and a group’s rejected updates are its own, so each group’s weights are solved from its own update counts: the result is the groups’ mean of their own reconstructions, as the groups’ EMAs average each its own group’s parameters, not an average of the outer model’s trajectory, which no profile follows.

def save_local(
step: int,
state: TrainState,
saved: bytes | None,
*,
share: DataPartition | None = None,
control: dict | None = None,
rung: JSON = None,
artifact: JSON = None,
boxed: Mapping[tuple[str, ...], LogicalAxes] | None = None,
rollout: Mapping[str, object] | None = None,
) -> None

Write state under step to this process’s local directory, asynchronously, with its rollout’s tree where it holds one (rollout, replicated: every process writes it whole).

The new step replaces the previous local step. As with save, a state with arrays in pinned host memory is written before this returns. The placement is saved with the state, so a resume onto a different placement raises before it reads shards from directories that do not hold them.

def rung(step: int) -> JSON

Return the rung of fit’s ladder that the state at step trained on.

The value is what save recorded, read from the directory restore reads the step from. It is None for a state saved outside fit, which trained on no ladder.

def boxed(step: int | str | None = None) -> dict[tuple[str, ...], LogicalAxes]

The axes the model at step boxed on its own parameters, as save recorded them; a checkpoint saved without them has none.

def variables(
*,
step: int | str | None = None,
ema: bool | None = None,
mesh=None,
layout=None,
param_dtype: jax.typing.DTypeLike | None = None,
parameter_roots: tuple[tuple[str, ...], ...] = PARAMETER_ROOTS,
unread: tuple[tuple[str, ...], ...] = (),
) -> Variables

Read the selected step’s live or averaged variables onto the requested layout.

Parameter storage conversion applies only to the owner’s parameter roots; other collections retain their recorded dtypes and placement. The subtrees at unread’s paths are neither placed nor read (dew._records.UNREAD).

def materialized(
template: Mapping[str, Variables],
placement: Mapping[str, Variables],
*,
step: int,
param_dtype: jax.typing.DTypeLike | None = None,
parameter_roots: tuple[tuple[str, ...], ...] = PARAMETER_ROOTS,
) -> Variables

The variables at step whose leaves template names (stored’s, under "variables", and under "ema" the averaged ones, merged over them), restored onto placement’s shardings, no other leaf read; the floating leaves under parameter_roots read as param_dtype where one is given, the rest in their stored dtypes. Every load of a run’s variables restores through it: a whole run’s onto its layout (variables) and a bank’s rows onto theirs (dew.inference.banks.CheckpointBanks).

def stored(step: int | str | None = None) -> Variables

Return what the checkpoint at step holds, without reading its values.

step defaults to the latest. Each state field comes back as a shape/dtype tree, and an unset field as None. A step its retention keeps for its model holds the model’s fields (its post-hoc EMA profiles, where it keeps them too, are restore_profiles’s); one kept for its profiles alone is refused.

def accumulation_template(step: int)

Return the persisted pending-array shapes, a DiLoCo run’s groups’ stacked, without reading them.

def position(step: int, share: DataPartition) -> bytes | None

The data position the reader of share resumes from at step (read_position), read alone, before any state is restored: a run opens its data there and reads its first batch while its state is still unplaced (Trainer.fit). None where the step saved no position.

def rollout_state(step: int, template: Mapping[str, object]) -> dict | None

The rollout’s tree the step at step holds (save’s rollout), read alone and placed as template’s leaves (jax.ShapeDtypeStructs) are; None for a step saved without one.

def restore(
template=None,
step: int | str | None = None,
*,
share: DataPartition | None = None,
served: Sequence[ProjectionGroup] = (),
)

Restore the state at step and the data position of share.

template is a pytree of jax.ShapeDtypeStruct naming the state leaves to restore. When a leaf has a sharding, the array is placed there, so a checkpoint written on one mesh restores onto whatever mesh this run uses. None restores every leaf as a host array. A template leaf that the checkpoint lacks raises an error naming it, unless the template holds it as a concrete array, which is then kept as it is.

A step that is the local one every process holds is read from the local directory, onto the placement it was written with; any other step is read from the persistent directory. The data position is returned as the bytes the reader of share resumes from (see read_position). Without a share, as for a caller that reads weights and no data, it is None.

served names the projection groups a step may hold packed where the template holds their members (Objective.served_groups): each such leaf, in any of the state’s trees, is read packed and split back into its members (dew.checkpoints.served.unpacked). Carried to 1.0, when it is deleted (ledger 1552).

def close() -> None

Wait for pending writes, then close the Orbax managers and the threads and metadata stores they hold, each even where another, or its own wait, failed: Orbax reports a failed asynchronous write once, from the wait, and its close still releases the manager. The first failure is raised with the rest noted on it. A closed Checkpoints reopens its managers on its next use.

def wait() -> None

Block until pending async writes have landed on disk.

Saving is async so it stays off the training loop’s critical path; anything that reads a checkpoint back has to call this first.

dataclass source

class Keep(
latest: int = 2,
every: int | None = None,
interval: datetime.timedelta | str | None = None,
where: Callable[[Kept], bool] | None = None,
weights_only: bool = False,
)

Keeps the union of the latest steps, periodic steps, wall-time-spaced steps and a predicate’s steps.

interval keeps checkpoints at least this much wall time apart; it does not keep every old checkpoint. where receives a retained-step record with its metrics. weights_only keeps the steps every, interval and where keep, beyond the latest, as their model alone (ROLES): the newest steps stay whole for a resume, and an archive of a long run holds no optimizer state it never resumes.