Skip to content

Train a diffusion model with Dew

This is part 2 of the series, and so I would strongly recommend going through Notebook 01 first. We carry on from the ideas we built there, the forward and reverse diffusion processes, noise schedules and samplers, and I will only recap them briefly here.

In the previous notebook, we built a diffusion model completely by hand: the noise schedule, the loss, the network and the samplers, every piece written from scratch so that we could see exactly what each one does. That is honestly the best way to understand how these models work, but you wouldn’t want to write all of it again every time you train a model. And that’s where Dew comes in. Each of those pieces already exists in Dew, so this time we can focus on the training run itself.

One training step. A clean flower x0 gets noise at a random level sigma to make x_t; the model sees x_t and sigma and guesses x0; the loss is how far the guess is from x0, and the update makes the next guess closer.1 noise a clean image2 guess it backx₀+σ ε, σ = 0.5=xₜDiT(xₜ, σ)guess x̂₀loss =w(σ) · ‖x̂₀ − x₀‖²update the weights
One training step. A clean flower x0 gets noise at a random level sigma to make x_t; the model sees x_t and sigma and guesses x0; the loss is how far the guess is from x0, and the update makes the next guess closer.1 noise a clean image2 guess it backx₀+σ ε, σ = 0.5=xₜDiT(xₜ, σ)guess x̂₀loss = w(σ) · ‖x̂₀ − x₀‖²update the weights

That’s one training step, and training is this step repeated a few tens of thousands of times.

In this notebook, we shall train a small diffusion transformer on Oxford Flowers at 128x128, look at how the loss behaves, generate flowers from pure noise, and then reload the trained weights from the checkpoint.

This installs Dew from GitHub with two of its extras: cuda13, which brings JAX’s CUDA 13 plugin along with the CUDA libraries it needs, and streaming, which brings Hugging Face datasets to download the flower images.

%pip install -q "dewml[cuda13,streaming] @ git+https://github.com/AshishKumar4/dew@71b43f2dc070cf289e1fbaf6177e484215d4ee1a" --extra-index-url https://pypi.dewml.dev/simple
install log
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 604.4/604.4 kB 2.7 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77.7/77.7 MB 23.1 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 532.5/532.5 kB 40.1 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 206.6/206.6 kB 18.0 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 91.4/91.4 kB 8.3 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.3/3.3 MB 87.7 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.9/53.9 MB 28.0 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 81.9 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 216.9/216.9 kB 20.2 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 137.7/137.7 kB 13.2 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 102.6/102.6 kB 9.2 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.0/8.0 MB 57.6 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.2/134.2 MB 15.9 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 89.9/89.9 MB 19.5 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56.6/56.6 kB 5.5 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 168.1/168.1 kB 16.5 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 32.1/32.1 MB 42.2 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 135.7/135.7 kB 13.2 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 87.5/87.5 kB 8.7 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.0/16.0 MB 52.4 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 109.2/109.2 kB 10.2 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 115.9/115.9 kB 11.3 MB/s eta 0:00:00
  Building wheel for dewml (pyproject.toml) ... done
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
cudf-cu13 26.6.0 requires pyarrow<24,>=19.0.0, but you have pyarrow 26.0.0 which is incompatible.
IMAGE_SIZE = 128
BATCH_SIZE = 64
STEPS = 30000
STAGES = 6
LEARNING_RATE = 2e-4
SAMPLE_STEPS = 40
RUN_DIR = "runs/02-diffusion"
SEED = 0
import jax
import jax.numpy as jnp
import numpy as np
from dew import figures
from dew.training import prepare_process
prepare_process() # Dew's setup for a training process, before the first JAX call
print(jax.devices())
[CudaDevice(id=0)]

A small helper to draw a batch of images in a grid. Dew keeps images as uint8 in [0, 255] when they come out of the data pipeline, but the model works in [-1, 1], so the helper accepts both.

from dew.artifacts import uint8_pixels
def show_images(images, columns=4):
images = np.asarray(images)
if images.dtype != np.uint8:
images = uint8_pixels(images)
return figures.images([images], columns=columns, size=440 / columns - 4)

We use the Hub copy pranked03/flowers-blip-captions, which carries a caption for each Oxford Flowers photo. We shuffle the photos again every epoch and keep going past the end, and Dew’s image preset resizes each photo to the size we want, flips and colour-jitters it so the model sees a little variety, and groups the photos into batches. A photo that doesn’t decode stops the run rather than being skipped, so we always know how many photos an epoch holds. train(DataPartition()) then gives us a stream of batches for this process.

read="memory" reads the whole split into memory. The very first run has to download the split from the Hub and prepare it, and provision=True is what allows that; later runs just reuse the prepared copy. And Loading(workers=0, threads=16) does all the reading right here in this process, with 16 threads, instead of in separate worker processes.

from dew.data import Hub, Loading, presets
from dew.data.compile import compile as compile_data
from dew.data.dataset import DataPartition
flowers = Hub("pranked03/flowers-blip-captions").shuffle(seed=0).repeat()
data = compile_data(flowers.pipe(presets.images, IMAGE_SIZE, batch=BATCH_SIZE, augment=True, rejects=False),
read="memory", provision=True,
loading=Loading(workers=0, threads=16, read_buffer=64))
print(data.records, "images,", data.steps_per_epoch, "steps per epoch")
stderr
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
WARNING:huggingface_hub.utils._http:Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
16:23:00 WARNING dew.data: preparing hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train (264.3 MiB) into /root/.cache/huggingface/datasets/pranked03___flowers-blip-captions/default/0.0.0/b45984ffbef8596a8b6ac2ac8959f2f8998049bb
6552 images, 102 steps per epoch
batch = next(iter(data.train(DataPartition())))
print(batch["image"].shape, batch["image"].dtype)
show_images(batch["image"][:16])
(64, 128, 128, 3) uint8
16 images

A diffusion model learns to undo noise. During training, we take a clean image x0x_0, pick a noise level, and make a noisy image

xt=αtx0+σtϵ,ϵ∼N(0,I)x_t = \alpha_t x_0 + \sigma_t \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)

where:

  • xtx_t is the noisy image at time tt,
  • x0x_0 is the clean image,
  • αt\alpha_t and σt\sigma_t are the signal and noise rates at time tt,
  • ϵ\epsilon is Gaussian noise.
One flower with more and more noise: x_t = x0 + sigma epsilon at eight of EDM's noise levels, from sigma 0.002, where it is clean, to sigma 80, where only noise is left.xₜ = x₀ + σ εσ = 0.002σ = 0.0352σ = 0.267σ = 1.29σ = 4.64σ = 13.7σ = 35σ = 80x₀, the clean image0.00280σ, log scale

The model sees xtx_t and the noise level and has to recover the clean image. Sampling then starts from pure noise at the highest noise level and removes it a little at a time, which is the reverse diffusion process we integrated with our samplers in notebook 01.

In notebook 01, we wrote the noise schedule, the model’s parameterization and the loss as separate pieces. Dew calls this combination of a noise schedule, what the network predicts and how the loss is weighted a Process, and it ships a few of them as presets (EDM, Karras, Cosine, Flow and Sqrt). We will use the EDM preset, which follows the paper “Elucidating the Design Space of Diffusion-Based Generative Models” by Karras et al. (2022).

Unlike the cosine schedule of notebook 01, which was variance preserving, this one is variance exploding: αt\alpha_t just stays at 1, and σ\sigma goes from 0.002 up to 80. Training draws its noise levels from a log-normal distribution, and sampling uses the paper’s ρ=7\rho = 7 grid, which spends more of its steps at low noise. The network also doesn’t predict the noise directly like our epsilon model did. The preset wraps it in the paper’s preconditioning, which scales the network’s input and mixes its output with the noisy image depending on σ\sigma.

EDM(regime="pixel") is the configuration, and calling it builds the Process. The regime picks the noise levels training draws: Karras et al.’s for images in pixel space, as here, or the higher, narrower ones of EDM2 for a model that denoises an autoencoder’s latents.

from dew.diffusion.presets import EDM
process = EDM(regime="pixel")()
schedule = process.sampler_schedule
times = process.times(8)
alphas, sigmas = schedule.rates(times)
print("sampling times:", np.round(np.asarray(times), 3))
print("alpha:", np.round(np.asarray(alphas), 3))
print("sigma:", np.round(np.asarray(sigmas), 3))
sampling times: [1.    0.857 0.714 0.571 0.429 0.286 0.143 0.   ]
alpha: [1. 1. 1. 1. 1. 1. 1. 1.]
sigma: [8.0000e+01 3.4992e+01 1.3699e+01 4.6370e+00 1.2870e+00 2.6700e-01
 3.5000e-02 2.0000e-03]

Here is one flower at each of those eight noise levels, from the most noise on the left to the least on the right. The images are clipped to [-1, 1] for display, so the leftmost ones look like plain static. And they really are: at σ=80\sigma = 80 the noise has a standard deviation of 80 while the pixels lie in [-1, 1], so there is basically nothing of the flower left to see.

image = batch["image"][0].astype(np.float32) / 127.5 - 1
noise = jax.random.normal(jax.random.key(SEED), image.shape)
noisy = [alpha * image + sigma * noise for alpha, sigma in zip(alphas, sigmas)]
show_images(np.clip(np.stack(noisy), -1, 1), columns=8)
8 images

In notebook 01, we built a UNet for our denoising model. This time we use a small diffusion transformer (DiT), the architecture from the paper “Scalable Diffusion Models with Transformers” by Peebles and Xie. It is a plain stack of transformer blocks conditioned on the noise level with adaLN-Zero. It cuts the 128x128 image into 4x4 patches, which gives us 1,024 tokens, runs them through twelve transformer layers, and returns one prediction per pixel. SimpleDiT from dew.nn.backbones is that transformer.

InputSpec(Field("image", (128, 128, 3))) tells the objective which batch field to generate and what shape it has. Note that this model has no conditions: it learns what flowers look like in general, but we can’t ask it for a particular one. We will get to that in notebook 03.

from dew import Field, InputSpec
from dew.nn.backbones import SimpleDiT
model = SimpleDiT(
patch_size=4,
emb_features=384,
num_layers=12,
num_heads=6,
output_channels=3,
dtype=jnp.bfloat16,
attention_impl="auto",
)
inputs = InputSpec(Field("image", (IMAGE_SIZE, IMAGE_SIZE, 3)))

DiffusionObjective computes the diffusion loss, the one we wrote by hand in notebook 01: it draws a noise level for each image, makes the noisy image, runs the model and weights the error. It also asks the trainer to keep an exponential moving average (EMA) of the weights. The EMA copy changes slowly and usually gives cleaner samples than the live weights, which is why we sample with it later. solver, guidance and steps only say how the objective samples when it previews images; they don’t change training at all.

Trainer does the rest: it computes the gradients, applies the AdamW optimizer, updates the EMA, and writes checkpoints to RUN_DIR. LocalTracker records every logged number in a JSON file, which we will plot after training.

from dew import Checkpoints, LocalTracker, Trainer
from dew.config import OptimConfig
from dew.objectives.diffusion import DiffusionObjective
from dew.sampling import EulerAncestral
objective = DiffusionObjective(
model, process, inputs,
ema_decay=0.999,
solver=EulerAncestral(), guidance=None, steps=SAMPLE_STEPS,
)
tracker = LocalTracker(f"{RUN_DIR}/tracking")
trainer = Trainer(
objective, OptimConfig(learning_rate=LEARNING_RATE),
key=jax.random.key(SEED),
checkpoints=Checkpoints(RUN_DIR),
tracker=tracker,
)
variables = jax.eval_shape(objective.init, jax.random.key(0))
n_params = sum(x.size for x in jax.tree_util.tree_leaves(variables["params"]))
print(f"{n_params / 1e6:.1f}M parameters")
35.5M parameters

fit trains until the step counter reaches STEPS, prints the loss every log_every steps, and writes a checkpoint at the end. We call it once for each of STAGES stages, each time with a later end, and keep the averaged weights after every stage. Each call carries on from the checkpoint the one before it wrote:

If RUN_DIR already holds a checkpoint, fit resumes from it, and the loop skips the stages already done.

done = Checkpoints(RUN_DIR).latest or 0
snapshots = []
for stage in range(1, STAGES + 1):
end = STEPS * stage // STAGES
if end < done:
continue # an earlier run of this cell trained past this stage
state = trainer.fit(data, steps=end, log_every=2000, checkpoint_every=STEPS)
snapshots.append((int(state.step), jax.device_get(state.averaged)))
Training SimpleDiT from step 0 to 5000: 35.5M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 64, float32 parameters, bfloat16 compute
step 2000/5000  loss 0.06594  data_wait_ms 0.03913  grad_norm 0.08774  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 15.58  step_time_ms 116.7  samples_per_sec 548.3  mfu 33.2%  hlo_flops_util 33.2%  hfu 33.2%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  0:05:46 left
step 4000/5000  loss 0.06934  data_wait_ms 0.03810  grad_norm 0.05747  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 15.58  step_time_ms 116.7  samples_per_sec 548.2  mfu 33.2%  hlo_flops_util 33.2%  hfu 33.2%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  0:01:57 left
step 5000/5000  loss 0.07218  data_wait_ms 0.03980  grad_norm 0.05083  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 15.58  step_time_ms 116.7  samples_per_sec 548.3  mfu 33.2%  hlo_flops_util 33.2%  hfu 33.2%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0
Trained 5000 steps in 0:10:40: first step after 52.41 s, then 8.5 step/s
91.5% of the wall time in steps, final loss 0.07218
Resumed from step 5000 in /content/runs/02-diffusion
Training SimpleDiT from step 5000 to 10000: 35.5M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 64, float32 parameters, bfloat16 compute
step 6000/10000  loss 0.06449  data_wait_ms 0.03747  grad_norm 0.04855  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.10  step_time_ms 117.0  samples_per_sec 547.1  mfu 33.1%  hlo_flops_util 33.1%  hfu 33.1%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  0:07:47 left
step 8000/10000  loss 0.07277  data_wait_ms 0.03687  grad_norm 0.04408  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.10  step_time_ms 116.7  samples_per_sec 548.2  mfu 33.2%  hlo_flops_util 33.2%  hfu 33.2%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  0:03:53 left
step 10000/10000  loss 0.07595  data_wait_ms 0.03689  grad_norm 0.04232  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.11  step_time_ms 116.7  samples_per_sec 548.3  mfu 33.2%  hlo_flops_util 33.2%  hfu 33.2%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0
WARNING:absl:Waiting for previous save to complete took 2.275977 seconds. If this number is high, consider checkpointing less frequently.
Trained 5000 steps in 0:10:13: first step after 27.25 s, then 8.5 step/s
95.5% of the wall time in steps, final loss 0.07595
Resumed from step 10000 in /content/runs/02-diffusion
Training SimpleDiT from step 10000 to 15000: 35.5M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 64, float32 parameters, bfloat16 compute
step 12000/15000  loss 0.06293  data_wait_ms 0.03695  grad_norm 0.03926  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.11  step_time_ms 116.8  samples_per_sec 547.9  mfu 33.1%  hlo_flops_util 33.1%  hfu 33.1%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  0:05:50 left
step 14000/15000  loss 0.06328  data_wait_ms 0.03735  grad_norm 0.03839  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.11  step_time_ms 116.7  samples_per_sec 548.5  mfu 33.2%  hlo_flops_util 33.2%  hfu 33.2%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  0:01:58 left
step 15000/15000  loss 0.05546  data_wait_ms 0.03948  grad_norm 0.03752  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.11  step_time_ms 116.7  samples_per_sec 548.5  mfu 33.2%  hlo_flops_util 33.2%  hfu 33.2%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0
WARNING:absl:Waiting for previous save to complete took 2.393647 seconds. If this number is high, consider checkpointing less frequently.
Trained 5000 steps in 0:10:13: first step after 27.08 s, then 8.5 step/s
95.5% of the wall time in steps, final loss 0.05546
Resumed from step 15000 in /content/runs/02-diffusion
Training SimpleDiT from step 15000 to 20000: 35.5M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 64, float32 parameters, bfloat16 compute
step 16000/20000  loss 0.07462  data_wait_ms 0.03753  grad_norm 0.03523  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.11  step_time_ms 116.9  samples_per_sec 547.4  mfu 33.1%  hlo_flops_util 33.1%  hfu 33.1%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  0:07:53 left
step 18000/20000  loss 0.06143  data_wait_ms 0.03911  grad_norm 0.03630  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.11  step_time_ms 116.7  samples_per_sec 548.6  mfu 33.2%  hlo_flops_util 33.2%  hfu 33.2%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  0:03:53 left
step 20000/20000  loss 0.07199  data_wait_ms 0.03837  grad_norm 0.03461  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.11  step_time_ms 116.7  samples_per_sec 548.6  mfu 33.2%  hlo_flops_util 33.2%  hfu 33.2%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0
WARNING:absl:Waiting for previous save to complete took 2.469201 seconds. If this number is high, consider checkpointing less frequently.
Trained 5000 steps in 0:10:15: first step after 28.85 s, then 8.5 step/s
95.2% of the wall time in steps, final loss 0.07199
Resumed from step 20000 in /content/runs/02-diffusion
Training SimpleDiT from step 20000 to 25000: 35.5M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 64, float32 parameters, bfloat16 compute
step 22000/25000  loss 0.06882  data_wait_ms 0.03734  grad_norm 0.03436  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.11  step_time_ms 116.7  samples_per_sec 548.3  mfu 33.2%  hlo_flops_util 33.2%  hfu 33.2%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  0:05:55 left
step 24000/25000  loss 0.06065  data_wait_ms 0.03622  grad_norm 0.03366  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.11  step_time_ms 116.6  samples_per_sec 548.8  mfu 33.2%  hlo_flops_util 33.2%  hfu 33.2%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  0:01:57 left
step 25000/25000  loss 0.07334  data_wait_ms 0.03719  grad_norm 0.03216  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.11  step_time_ms 116.6  samples_per_sec 548.7  mfu 33.2%  hlo_flops_util 33.2%  hfu 33.2%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0
WARNING:absl:Waiting for previous save to complete took 2.411053 seconds. If this number is high, consider checkpointing less frequently.
Trained 5000 steps in 0:10:16: first step after 29.96 s, then 8.5 step/s
95.1% of the wall time in steps, final loss 0.07334
Resumed from step 25000 in /content/runs/02-diffusion
Training SimpleDiT from step 25000 to 30000: 35.5M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 64, float32 parameters, bfloat16 compute
step 26000/30000  loss 0.06060  data_wait_ms 0.03639  grad_norm 0.03320  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.11  step_time_ms 117.0  samples_per_sec 546.9  mfu 33.1%  hlo_flops_util 33.1%  hfu 33.1%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  0:07:47 left
step 28000/30000  loss 0.05745  data_wait_ms 0.03873  grad_norm 0.03192  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.11  step_time_ms 116.8  samples_per_sec 548.2  mfu 33.2%  hlo_flops_util 33.2%  hfu 33.2%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  0:03:50 left
step 30000/30000  loss 0.06163  data_wait_ms 0.03767  grad_norm 0.03184  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.11  step_time_ms 116.8  samples_per_sec 548.1  mfu 33.2%  hlo_flops_util 33.1%  hfu 33.1%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0
WARNING:absl:Waiting for previous save to complete took 2.813380 seconds. If this number is high, consider checkpointing less frequently.
Trained 5000 steps in 0:10:16: first step after 29.53 s, then 8.5 step/s
95.1% of the wall time in steps, final loss 0.06163

The tracker wrote one line per log step to scalars.jsonl, here every 2,000 steps and at the end of each stage. By step 2,000, the first point we logged, the loss is already down to about 0.07, and after that it jumps around between about 0.055 and 0.076 without falling much further. Don’t worry, that is expected: every batch draws new noise levels, and some noise levels are much harder to denoise than others. The model keeps improving long after the loss flattens out; the noise hides it.

import json
rows = [json.loads(line) for line in open(f"{RUN_DIR}/tracking/scalars.jsonl")]
rows = [row for row in rows if "train/loss" in row["scalars"]]
figures.curve({"loss": [row["scalars"]["train/loss"] for row in rows]},
x=[row["step"] for row in rows], x_label="step", y_label="loss")
loss against step0.060.070.08100002000030000steploss0.06

To generate an image, we run the noise process backwards, just like we did with our samplers in notebook 01. process.noise draws the starting point, pure noise with the spread of the highest noise level. process.denoiser wraps the model and its weights into the function a sampler calls: given a noisy image and its time, it returns the model’s guess of the clean image and of the noise. sample then runs the solver over SAMPLE_STEPS noise levels in one compiled loop.

The solver is EulerAncestral, the Euler-A sampler we talked about in notebook 01: Euler’s method on the reverse diffusion SDE, so a bit of fresh noise goes back in at every step.

state.averaged is the weights tree with the EMA weights in place of the live ones, and objective.model_variables keeps only the model’s own weights.

from dew.sampling import sample
def generate(params, count, key):
denoise = process.denoiser(model, objective.model_variables(params), {})
x_T = process.noise(key, (count, IMAGE_SIZE, IMAGE_SIZE, 3))
return sample(denoise, x_T, SAMPLE_STEPS, solver=EulerAncestral(), key=key)
images = generate(state.averaged, 32, jax.random.key(1))
show_images(images[:16])
16 images

Most of these are clearly flowers, with petals, a centre and leaves behind them, but they are still soft and painterly, and one or two are more blob than flower.

Remember the stages? Here is the same starting noise, sampled with the averaged weights we kept after each one. Every frame starts from exactly the same xTx_T and draws the same fresh noise along the way, so whatever changes from frame to frame is the model.

frames = [uint8_pixels(generate(params, 8, jax.random.key(1)))
for _, params in snapshots]
figures.images(frames, captions=[f"step {step:,}" for step, _ in snapshots], columns=4,
size=96)
8 images at each of 6 pointsstep 5,000step 10,000step 15,000step 20,000step 25,000step 30,000

By step 5,000 every image already holds a flower-shaped patch of the right colours on green. What the later stages add is detail. The pink flower at the top left gets separate petals and a yellow centre by step 10,000, and the one at the bottom right turns from pink to red over the last three stages. The third in the top row and the first in the bottom row never quite come into focus.

The checkpoint in RUN_DIR holds the arrays: the weights, the EMA, the optimizer state and the step counters. It does not hold the code. So to reload, we build a new Trainer over the same objective and call place(), which restores the latest checkpoint. Sampling with the same key gives the same images, which shows that the weights came back unchanged.

restored, _, _ = Trainer(
objective, OptimConfig(learning_rate=LEARNING_RATE).build(STEPS),
key=jax.random.key(SEED),
checkpoints=Checkpoints(RUN_DIR),
).place()
print("restored step:", int(restored.step))
again = generate(restored.averaged, 32, jax.random.key(1))
print("largest difference from the samples above:", float(jnp.max(jnp.abs(again - images))))
Resumed from step 30000 in /content/runs/02-diffusion
restored step: 30000
largest difference from the samples above: 0.0

Notebook 03 trains on the captions as well, so that you can ask for a flower in words. Notebook 04 loads this notebook’s checkpoint and compares Dew’s samplers on it. And if you would rather train from the command line, recipes/diffusion/train.py runs the same kind of training.