Skip to content

Dew documentation

Dew is a library for training neural networks with JAX and Flax. A training run is a Flax model, an Objective that defines the variables and the loss, a Dataset that yields batches, and a Trainer that runs the optimization on one device or a mesh of devices. Dew ships objectives for language models, image and video diffusion, diffusion language models, JEPA encoders and post-training (SFT, DPO, GRPO, PPO).

import itertools
import jax
import jax.numpy as jnp
import numpy as np
import optax
from flax import linen as nn
from dew import Trainer
from dew.data.dataset import Loader
from dew.objectives.base import Aux, Objective
class Regression(Objective):
def __init__(self, model):
self.model = model
def init(self, key, variables=None):
return self.model.init(key, np.zeros((1, 1), np.float32))
def loss(self, variables, batch, step):
errors = (self.model.apply(variables, batch["x"]) - batch["y"]) ** 2
return self.row_mean(errors, batch), Aux(metrics={})
x = np.linspace(-1, 1, 32, dtype=np.float32).reshape(32, 1)
batch = {"x": x, "y": 2 * x + 1}
data = Loader(train=lambda partition: itertools.repeat(batch), val=None, records=32, batch=32)
trainer = Trainer(Regression(nn.Dense(1)), optax.sgd(0.1), key=jax.random.key(0))
state = trainer.fit(data, steps=100, log_every=50)

The pages assume you know Python, NumPy and the basics of training with gradients. Pages that also need Flax or sharding say so at the top.

  1. Installation: install Dew and check which devices JAX sees.
  2. Quickstart: the example above, one part at a time.
  3. Key concepts: the model, the objective, the dataset and the trainer.
  4. Tutorials: notebooks that train diffusion models, a language model and a JEPA encoder on real data.
TopicPage
Batches, dataset specifications, TFDS and Hugging Face data, resuming the data streamTraining data
Writing an ObjectiveCustom objectives
Validation metrics, previews and experiment trackersEvaluation and tracking
Saving, resuming and exportingCheckpoints
Diffusion processes, presets, solvers and guidanceDiffusion processes and solvers
Training an image diffusion modelDiffusion training
Language models: training, loading published checkpointsLanguage models
Diffusion language models: masked diffusion, DiffusionGemmaDiffusion language models
Mixture-of-experts layersMixture of experts
SFT, DPO, GRPO and PPOPost-training
Text generation and servingGeneration and serving
JEPA encodersJEPA
Meshes, sharding rules and several hostsDistributed training, Multiple hosts, Cloud TPUs
Checkpoints Dew can loadSupported models
Every public class and functionAPI reference

Dew is research software before version 1.0, so the API and checkpoint formats can change between versions. I have run it on CPU, on pools of local processes, on single GPUs, on one host with four GPUs and on one TPU v6e chip, but not on two physical nodes. Multiple hosts lists what has and has not been run. The roadmap lists work still to complete or qualify.

Papers and attribution lists the papers and upstream code behind the models and methods. How Dew works follows one training run from a model to kernels on the GPU, and holds the shared contracts. The binding training, data, post-training, testing, algebraic optimization and kernel templates specs stay in the repository under docs/design. Research notes stay under docs/research.