Skip to content

Step benchmarks

tools/benchmark_step.py times one complete compiled optimization step through Trainer for each architecture, and tools/benchmark_data.py times input loading on its own. The tables below record past runs. Each states its hardware, source revision and shapes, and none promises the same throughput on the current checkout.

The FLOP counts come from the optimized HLO of the compiled executable, through dew.telemetry.instrumentation.compiled_flops. It counts every dot and convolution from its own shapes, along with the cuBLAS matmul, cuDNN convolution and cuDNN fused-attention custom calls that a GPU backend turns them into. util is the number the trainer logs as train/hfu: the step’s measured FLOPs divided by the step time and by the dense bf16 peak of one device (97.5 TFLOP/s for the RTX 4080). It counts the hardware’s work, a remat’s recomputed forward included. train/mfu divides the objective’s analytical training work instead (Objective.training_flops): the model’s forward and backward on the batch’s real tokens, with causal attention halved and without the optimizer, loss bookkeeping or recomputation. No HLO count stands in for it. tools/reference_runs/work.py writes the same count for a reference run in another framework, so tools/benchmark_torch.py’s MFU shares the trainer’s numerator.

ColumnMeaning
ms/stepTime per step of the loop as a run dispatches it.
p10 / p50 / p90 msPercentiles from a second window of the same length that waits on every step, so a long tail shows up there.
GFLOP/stepFLOPs of the compiled step, counted from the optimized HLO.
utilMeasured FLOPs over step time and device peak (train/hfu).
mfuThe objective’s training FLOPs over step time and the mesh’s peak (train/mfu); n/a where the model has no count or the device no peak. Tables recorded before 2026-10-16 lack it.
peak GiBPeak device memory, measured: each device’s allocator high-water over the case’s run, native workspaces included, read in a process of the case’s own that loads the step from a compilation cache and compiles nothing; the largest device’s. Tables recorded before plan GiB was a column read the high-water in the process that compiled the step, so their peak can include the compile’s own.
plan GiBThe bytes XLA planned for the step: its arguments, outputs and temporaries, without native workspaces.
compile sTime to compile the step.
python tools/benchmark_step.py --preset small --architectures unet --json-out bench.json

jax 0.11.1 / jaxlib 0.11.1 (CUDA), flax 0.12.9, optax 0.2.8, driver 595.84, RTX 4080 16 GiB, Dew at 0653e64. Every model ran in bf16 (dtype=bfloat16) on a single device, with MeshSpec(fsdp=1), adam, 2 warmup steps and 100 measured steps. Each architecture ran in its own invocation (--architectures unet and so on), so each row’s peak memory is that architecture’s alone. The host was otherwise idle.

architecturesamplebatchparamsms/stepp10 / p50 / p90 mssamples/sGFLOP/steputilpeak GiBcompile s
unet64x64x31610,159,29916.418.2 / 18.4 / 19.0977.2646.440.5%0.7536.4
uvit64x64x31624,351,02423.023.9 / 24.1 / 25.1695.7617.827.6%1.719.4
simple_udit64x64x31623,381,4249.410.4 / 10.9 / 12.11696.4363.139.5%1.0510.5
simple_dit64x64x31619,835,5687.98.8 / 9.1 / 10.02036.0292.938.2%0.909.9
simple_mmdit64x64x31636,385,58412.914.7 / 15.4 / 18.41240.0383.730.5%1.4016.3
hierarchical_mmdit64x64x31655,498,18832.738.1 / 38.8 / 39.2489.3737.423.1%3.3836.9
hybrid_dit64x64x31619,344,0488.99.6 / 10.0 / 10.61795.5244.628.2%0.8613.1
video_dit8x64x64x3425,155,50417.318.3 / 18.7 / 19.6231.8760.045.2%1.5912.6
unet_3d8x64x64x3411,045,69933.936.7 / 37.2 / 39.6117.81384.141.8%1.7246.6
jepa_encoder64x64x31612,149,5689.410.6 / 10.8 / 11.11699.1297.332.4%0.7013.7
jepa_video_encoder8x64x64x3416,143,36018.320.1 / 20.3 / 22.9218.2758.142.4%1.2820.5
causal_transformer512 tokens1666,950,78483.083.7 / 83.8 / 84.0192.73406.442.1%5.8110.1

The small preset on current main also has unet_2d_condition, sd3_transformer, flux_transformer, multimodal_transformer and diffusion_gemma cases, which have no rows here. jepa_predictor has no step of its own, because it trains inside the two JEPA rows. The causal_transformer row has the width of GPT-2 small, three layers and a 50k vocabulary. At this revision most of its FLOPs were the tied fp32 vocabulary projection and its gradients, run as cuBLAS custom calls.

Rows taken while something else ran on the host came out much slower; unet read 54 ms/step under load against 16.4 idle. In the video rows the p90 is 6-13% above the p50. That spread comes from the host scheduler, and the step itself is steady.

From the table:

  • simple_dit, simple_udit and hybrid_dit each run under 10 ms/step at 28-40% of the card’s dense bf16 peak at these shapes.
  • unet does the most arithmetic for its time of the image models. 646 GFLOP/step in 16 ms is 40.5% of peak, ahead of the transformers at the same resolution. With XLA’s own cost_analysis() as the numerator, the same measurement shows 28.7 GFLOP/step, and the gap is convolution arithmetic that cost analysis cannot see.
  • unet_3d is the slowest diffusion step in the table (33.9 ms/step, 41.8% of peak). Its 3D convolutions do 1,384 GFLOP/step, 1.8 times the 760 of video_dit on the same (8, 64, 64, 3) samples, and video_dit’s step (17.3 ms) takes about half as long.
  • hierarchical_mmdit is the largest diffusion model here (55 M; the 67 M causal_transformer is a language model) and has the slowest image step (32.7 ms), which fits its 1024-token finest stage.
  • Compile time dominates a short run, at 9-47 s per architecture against 8-84 ms per step, which is why a run keeps the persistent compilation cache by default (Compilation cache); these compile times are cold compiles, with the cache off.
JAX_PLATFORMS=cuda XLA_PYTHON_CLIENT_PREALLOCATE=false XLA_PYTHON_CLIENT_MEM_FRACTION=0.8 \
python tools/benchmark_step.py --preset small --architectures <arch> --json-out <arch>.json

Same card, driver and library versions, with Dew at fd433e7, the tree before the cudnn padding of 92a591c. The simple_mmdit, hierarchical_mmdit and unet rows after that padding are in Performance measurements. The host was otherwise idle, with one process per architecture, 2 warmup steps and 100 measured steps. Each parameter count is 99,840 higher than in the first table, because the condition encoder’s table (CharTable, 130 by 768) is in the state tree at this revision; at 0653e64 it was a constant in the executable. The second column repeats the 09-02 ms/step of the four rerun architectures.

architecture09-02 ms/stepms/stepp10 / p50 / p90 mssamples/sGFLOP/steputilpeak GiBcompile s
simple_dit7.97.027.5 / 7.6 / 7.92278.3292.942.8%0.838.0
hierarchical_mmdit32.733.9537.3 / 37.6 / 39.3471.2737.422.3%3.5050.4
video_dit17.317.0618.5 / 18.9 / 20.3234.5760.045.7%1.4113.0
causal_transformer83.088.7889.5 / 89.7 / 89.9180.23406.439.4%4.519.5

The decoder is 7% slower than on 09-02, for two reasons. The first is the chunked vocabulary head, which landed after the 09-02 table; at its default of four chunks it costs 1.9 ms against the full-vocabulary pass. Measured in the same tree with --cases setting head_chunks, 50 steps each, 1 chunk takes 87.02 ms and 5.67 GiB, 2 chunks 88.39 ms and 4.85 GiB, 4 chunks 88.90 ms and 4.51 GiB, and 8 chunks 89.67 ms and 4.41 GiB. So the default gives up 2.2% of the step to save 1.2 GiB. The other 3.8 ms are in the decoder itself. Rerun the same day on the same card, 0653e64 reads 83.26 ms, and the tree at fd433e7 with one head chunk reads 87.02. That difference comes from the decoder’s changes between the two commits, and I did not measure it further.

The preset’s MoE causal_transformer case (8 experts, top-2 on every second layer) does not run under XLA_PYTHON_CLIENT_PREALLOCATE=false. Its step asks for one 4.5 GiB buffer, and the BFC allocator, growing on demand into a 12.8 GiB budget, cannot place it (RESOURCE_EXHAUSTED ... 4.47GiB). The same tree runs with the default preallocation. The full-vocabulary decoder at 0653e64 fails the same way (4.80 GiB). On this card, benchmark a step that needs a single buffer over about 4.5 GiB with the default preallocation.

The same executables, counted both ways on 2026-09-02, with one compile each:

architecturecost_analysis() GFLOPoptimized HLO GFLOPratio
unet28.7646.422.50x
unet_3d145.81384.19.50x
causal_transformer1320.13406.42.58x
uvit327.4617.81.89x
video_dit693.3760.01.10x
hybrid_dit223.7244.61.09x
jepa_video_encoder698.9758.11.09x
simple_mmdit355.4383.71.08x
jepa_encoder281.5297.31.06x
hierarchical_mmdit715.2737.41.03x
simple_udit360.8363.11.01x
simple_dit296.9292.90.99x

cost_analysis() misses the arithmetic that the backend moves into its own kernels. For the two UNets that is the convolution custom calls, and for the decoder it is six cuBLAS calls for the tied fp32 vocabulary head and its gradients (2.58x); uvit has a mix of both. The pure-transformer rows agree to within a few percent in either direction, and that difference is the elementwise work, which cost_analysis() counts and the matmul count leaves out. The one row below 1.0 (simple_dit at 0.99x) comes from that elementwise accounting on top of the matmuls, and no kernels are missing there. How large these ratios come out depends on what XLA keeps visible, and XLA chooses differently between recompiles of the same code, while the HLO count stays the same. This agrees with the audit in docs/research/benchmark-parity.md, which found 22.50x, 2.372x and 0.987x for the three architectures it counted.

JAX_PLATFORMS=cpu XLA_FLAGS=--xla_force_host_platform_device_count=8 \
python tools/benchmark_step.py --preset cpu-smoke --steps 2

This preset runs tiny models on a simulated 8-device CPU mesh. It checks the tool itself and says nothing about the hardware; tests/test_benchmark_step.py runs one of its cases so that the tool keeps working with the trainer internals it drives. Utilisation and peak memory come back null, because a CPU has no published peak FLOP/s and no allocator counter.

python tools/benchmark_data.py data:oxford-flowers --batch 8 \
--data.image-size 64 --steps 100 --warmup 5 --loading.workers {0,8} \
--data.source <prepared version directory>

The dataset was Oxford Flowers 102 from local TFDS ArrayRecord files: 8189 records, resized to 64px, with flip and jitter augmentation and CLIP tokenization per record. The measurement predates the chain API, and the command above is the one it ran. Today the tool times a run’s chain, python tools/benchmark_data.py --steps 100 --data run.py, here the images preset over a prepared TFDS source with augment=True and its captions; preparation remains separate.

grain workerssamples/sp50 stepp95 step
0 (in-process)322.125.1 ms32.8 ms
8505.00.05 ms77.1 ms

With workers, the p50 is a queue read, so the loader only shows up in the p95. At 8 workers the pipeline delivers 505 images/s. The image rows of the first step table consume 489-2036 samples/s, and all but hierarchical_mmdit (489) are above 505. So at 64px this loader keeps up with hierarchical_mmdit and starves the other image models; if an image run shows a low train/mfu, check the loader first. The video rows count 8-frame clips, and this image loader does not measure video throughput.

These two points are not the loader’s ceiling. The measured run read with 16 threads. tools/benchmark_data.py times a chain under its own loading, as a run reads it; --loading.workers and --loading.threads set a Loading the compile takes over the chain’s own loading(...) or threads(n), though a chain’s explicit workers(n) stage still runs as written. Oxford Flowers is also only 8189 small records, far from a sharded 12M-record set.