Skip to content

Key concepts

A Dew training run is built from four objects, each responsible for one part of the work:

ObjectResponsible forWritten by you when
Model, a flax.linen.ModuleThe forward computation and the shapes of its variablesYou need a new architecture. The built-in ones are classes in dew.nn.backbones.
ObjectiveInitializing the variables, the loss, evaluation outputs, and which weights keep a moving averageYou need a loss Dew does not have.
DatasetThe records, from a source through your steps and a preset, in batchesAlways: a source and a preset, plus any steps of your own.
TrainerThe device mesh, the compiled step, the optimizer update, the moving average, checkpoints and loggingNever; it is configured, not subclassed.

Trainer.fit returns a TrainState, the numerical state of the run: the variables, the optimizer state, the root random key, the step counters and the moving-average copy. Continuing a run also needs the data position, which checkpoints store beside the state, and the configuration that built the objective and trainer (a recipe writes it to run.json).

This trains a small decoder on one repeated sentence and generates from it. It downloads nothing and runs on a CPU.

import jax
import numpy as np
from dew import Trainer
from dew.optim import OptimConfig
from dew.data import ByteTokenizer, Memory
from dew.nn.backbones import CausalTransformer
from dew.objectives.lm import LMObjective
from dew.sampling import Sampling, generate
tokenizer = ByteTokenizer()
text = tokenizer.encode("dew trains jax models. " * 3)
rows = np.tile(np.asarray(text[:65], np.int32), (8, 1))
data = Memory({"text": rows}).repeat().batch(8)
model = CausalTransformer(
vocab_size=tokenizer.vocab_size,
emb_features=64, num_layers=2, num_heads=4,
mlp_features=256, max_seq_len=128)
objective = LMObjective(model, seq_len=64)
trainer = Trainer(objective, OptimConfig(learning_rate=3e-3),
key=jax.random.key(0))
state = trainer.fit(data, steps=100, log_every=25)
prompt = [tokenizer.encode("dew")]
out = generate(model, state.variables, prompt, max_new_tokens=40,
key=jax.random.key(1),
sampling=Sampling(temperature=0))
print(tokenizer.decode(out.tokens[0]))

Output on two cores of a shared workstation CPU:

Training CausalTransformer from step 0 to 100: 147,840 parameters, on 1 × cpu, batch 8, float32
step 25/100 loss 0.03061 ce 0.03061 perplexity 1.031 token_accuracy 100.0% step_time_ms 23.68 samples_per_sec 337.9 accepted 100.0% 0:00:01 left
step 50/100 loss 0.01018 ce 0.01018 perplexity 1.010 token_accuracy 100.0% step_time_ms 24.52 samples_per_sec 326.2 accepted 100.0% 0:00:01 left
step 75/100 loss 0.006710 ce 0.006710 perplexity 1.007 token_accuracy 100.0% step_time_ms 24.68 samples_per_sec 324.1 accepted 100.0% 0:00:01 left
step 100/100 loss 0.005113 ce 0.005113 perplexity 1.005 token_accuracy 100.0% step_time_ms 22.84 samples_per_sec 350.2 accepted 100.0%
Trained 100 steps in 0:00:04: first step after 2.06 s, then 42.6 step/s
53.0% of the wall time in steps, final loss 0.005113
dew trains jax models. dew trains jax model

This is the output with stdout piped to a file; on a terminal, fit draws the same numbers as one live panel with a progress bar and a sparkline per metric. Each step line shows the loss and the objective’s metrics of that step (for LMObjective: cross entropy, perplexity and token accuracy), and the step time and throughput over the interval since the previous line. Each row holds 65 byte ids. LMObjective(seq_len=64) feeds the first 64 to the model and scores its predictions of the 64 that follow, shifted by one position. generate returns the prompt followed by the new tokens; temperature=0 picks the most likely token at every step.

A model is a flax.linen.Module with init and apply. Its variables are a nested dictionary of arrays, and it has no knowledge of training. Build one from its class, from dew.nn.backbones import CausalTransformer. A saved run records the class by its import path, and a configuration file may name it by a short alias (causal_transformer) instead.

Your own model needs nothing to be recorded: its import path names it, and its constructor fields are its record. Put it in a module, here mymodels.py:

import flax.linen as nn
class ResidualMLP(nn.Module):
"""A denoiser `DiffusionObjective` can train: it maps a noisy sample, the
noise level's embedding and optional text context to its prediction."""
features: int = 64
@nn.compact
def __call__(self, x, temb, textcontext=None, train=False):
return x + nn.Dense(x.shape[-1])(nn.gelu(nn.Dense(self.features)(x)))

Every checkpoint a run of this model writes records "architecture": "mymodels:ResidualMLP" and "config": {"features": 64}, and TextToImage.from_run and dew.pipeline rebuild the model from that record in any process that can import mymodels. A decoder’s run loads the same way through TextGeneration.from_run and Pretrained.from_run. A composite model records each part inside its own record, and an adapted model records its base model with the adapter’s rank, alpha and modules.

A package outside Dew works the same way, with nothing to register: a field typed with a class takes the record {"class": "mypackage.layers:Gate", "fields": {...}} of any class that derives from it or function declared to return it, and a function as {"function": "mypackage.layers:gate"}. Where a person writes a record, the class may be one of Dew’s short aliases instead ("simple_dit", "mla"), which dew.registry lists by kind.

A package can bring a whole kind of run the same way. Its own RunConfig subclass builds its objective and data in prepare, and run.json names that class, so dew train runs/x/run.json --trust mypackage trains it again in any process. Its objective may score into a dataclass of its own (dew.artifacts.Artifact), which its metric reads by declaring reads = MyArtifact; the trackers show previews of Dew’s artifact types, so a package shows its own as one of them, a spike raster as an ImageGrid. Its modules may name logical axes of their own (nn.with_logical_partitioning(init, ("inputs", "neurons"))), which the run places by adding rules to its layout, Layout(rules=(*DEFAULT_RULES, ("neurons", "fsdp"))). A record arriving from elsewhere imports a package’s modules only once the reader trusts the package (trust=("mypackage",), --trust mypackage), as trust_remote_code does in transformers.

A model defined where no import path reaches it, in __main__ or inside a function, trains, checkpoints and resumes the same way. The first checkpoint warns, and loading the run says, that its model has no import path.

A model written in Flax NNX trains through Flax’s own bridge, flax.nnx.bridge.ToLinen, which turns an NNX module into a Linen one with its parameters, its other state (a BatchNorm’s batch_stats) and its random streams (the dropout stream an objective passes to apply). The bridge runs the module’s __call__, which is all DiffusionObjective calls on a denoiser: bridge.ToLinen(MyDenoiser, args=(channels,)) is the model. LMObjective reads a model’s logits by its logits method where it has one and by its own forward otherwise (dew.nn.protocols.forward_method), so an NNX language model whose __call__ returns its logits trains through the bridge too; a Linen class over the bridge only states what LMObjective asks of a token model beside them, such as causal:

from flax.nnx import bridge
class MyTokens(bridge.ToLinen):
causal: bool = True
model = MyTokens(MyNNXLanguageModel, args=(vocab_size,))

Such a model checkpoints and resumes as any unregistered model does. LoRA cannot adapt it: the adapter intercepts flax.linen.Dense calls, and an NNX Linear behind the bridge makes none, so LoRA.apply refuses an NNX model rather than train factors the forward never reads. Adapt it in NNX before the bridge instead, with flax.nnx.LoRALinear in place of nnx.Linear or flax.nnx.LoRA beside it.

Dew’s modules name the logical axes of their parameters, such as embed, heads and mlp. The trainer maps those names onto the device mesh, so the model code does not change when the mesh does. Distributed training describes the mapping.

An objective implements two methods:

  • init(key, variables=None) returns the model’s variables.
  • loss(variables, batch, step) returns the loss as a Ratio(total, mass) and an Aux with metrics to log.

It can also implement evaluate for validation and preview for samples, and it names the weights that keep an exponential moving average (EMA) in its ema attribute. Its shown attribute maps metric names to Shown values that tell the training display how to show them: Shown(better="higher", percent=True), for an accuracy, colours a rise as progress and prints the value as a percentage.

Dew ships objectives for autoregressive language modeling (LMObjective), image and video diffusion (DiffusionObjective), masked and block diffusion over tokens (MaskedDiffusionObjective, BlockDiffusionObjective), JEPA (JepaObjective), preference and reinforcement learning (DPOObjective, GRPOObjective, PPOObjective, FlowGRPOObjective) and distillation (DistillationObjective). A new kind of model is a new module; a new kind of training is a new objective. Custom objectives writes one.

A Dataset is one chain: a source (Memory, Files, Hub or TFDS), the steps that read its records, and a preset or batch(n) that ends it. Trainer.fit compiles it into a Loader of two functions: train(partition) opens training batches and val(partition) opens one finite validation pass. DataPartition names the process’s share of the global batch. The example’s chain is a Memory source over column arrays, repeated and batched, and checkpoints save its consumed position.

The presets in dew.data.presets prepare one kind of record for a model and end in batch: images arrive as uint8 arrays in [0, 255] and token windows as int32 IDs under "text". A custom order writes the chain’s verbs itself. Training data covers read modes, storage, mixtures and exact resume.

Dataset to global batch: train(partition) opens an iterator of host batches, and shard_batch assembles each into one jax.Array split over the mesh's batch axes. Dataset to global batch: train(partition) opens an iterator of host batches, and shard_batch assembles each into one jax.Array split over the mesh's batch axes.

Trainer(objective, optimizer, key=...) takes an OptimConfig or an Optax optimizer, and a JAX random key or integer seed. fit(dataset, steps=...) builds a configured optimizer over the run’s length, places the variables on the devices, compiles one training step and runs it until the step counter reaches steps. Between steps it logs every log_every steps, evaluates every eval_every steps, and writes a checkpoint every checkpoint_every steps when the trainer was given checkpoints=Checkpoints(directory).

Trainer.fit: the state is placed on the mesh, the dataset's iterator feeds a prefetcher, and each compiled step runs the loss, the gradient, the optimizer update and the EMA update; logging, evaluation and checkpoints run on the host between steps. Trainer.fit: the state is placed on the mesh, the dataset's iterator feeds a prefetcher, and each compiled step runs the loss, the gradient, the optimizer update and the EMA update; logging, evaluation and checkpoints run on the host between steps.

The same call runs on one device or many. Trainer(..., mesh=MeshSpec(fsdp=4)) shards the parameters and optimizer state over four devices; the objective, the model and the data do not change.

The trainer logs the loss, the objective’s metrics, an injected learning rate and train/grad_norm: the mean global norm of the gradient each committed update hands the optimizer, before any clipping. A compiled step returns it as Committed.norm over Committed.count.

TrainState keeps three counters. step counts attempts, and together with the root key it determines the next random draw. microstep counts accepted microbatches. updates counts optimizer updates, which differs from microstep when gradients are accumulated. state.variables holds the live variables, and state.averaged holds the same tree with the EMA weights in place.

After training, objective.pipeline(state) wraps the weights in an inference task, such as text generation or text-to-image, and Pretrained.save writes a checkpoint in its source’s own format. Checkpoints lists which artifact continues what.