dew.telemetry.benchmark
Wall-clock timing of calls: samples, which every benchmark tool in Dew measures independent
calls with, carried, which times calls that carry a state from one to the next, benchmark,
which lays out many calls’ timings as a table, and compared, which reads two calls against each
other by their paired ratio; a GPU call’s own kernels’ device time, as CUPTI reads it
(device_time), which the kernel tuner and the kernel benchmarks read; and the process’s resident
memory as the kernel reports it (resident_memory), which the memory benchmarks read.
A run is timed from the call to the moment its output is ready on the devices
(jax.block_until_ready), so an asynchronous dispatch is timed as the work it
starts, and reading the output back to the host is not timed. The warm-up runs,
which compile what the call traces, are not kept. The caller passes what is
timed, a jitted function bound to its inputs: nothing is jitted here.
| Name | Summary |
|---|---|
EVICTION_BYTES | The bytes an eviction reads (eviction, behind): over twice the L2 of every GPU Dew targets (an H100’s 50 MB, a B200’s 126 MB), so a read of them leaves none of a timed call’s operands in it. |
Benchmark | What benchmark measured: each call’s timed runs in seconds, in order, and its last output. |
Comparison | A call’s paired ratios to its reference judged (judged): each pair’s ratio of the call’s seconds to the reference’s, the median ratio and its interval (_median_rank), and the reference against itself (its A/A), whose interval is the placement’s own noise: a difference is one where interval lies apart from 1 and outside alone (differs). |
CuptiHeld | CUPTI’s one subscription is another subscriber’s, a profile capture’s or Nsight’s, so no device time is read (device_time): the kernel tuner keeps a launch’s derived config until a later compile, and a benchmark reports it. |
DeviceTime | The device time of each GPU call device_time’s session reads: its own kernels’ as CUPTI times them, from their start to their end on the device, which no host dispatch or other process’s work enters. |
KernelTime | One kernel a call launched and its device time, as CUPTI read it (DeviceTime.kernels). |
NoKernels | A call CUPTI read none of its own kernels of (DeviceTime), whose time would read as zero. |
behind | The program that runs call(*args) behind an eviction, a read of evicting (eviction): called as (evicting, args), the call waits for the read (an optimization barrier) and both outputs are returned, so the compiler keeps it. |
benchmark | Time each of calls, in order, repeats times after warmup untimed runs (samples), and return the runs’ seconds and each call’s last output (Benchmark); no two of a call’s outputs are held at once. |
carried | count calls of step, each handed the state the last returned (a training step’s state, a decoder’s cache), and the seconds they took, with the state the last call returned. |
compared | first against second, each timed as samples times one call, in pairs interleaved pairs whose order a fair coin picks each pair (a call that runs second in a pair runs on whatever the first left), and second against itself in as many pairs, read as the median of each pair’s ratio of seconds and the interval its pairs’ order statistics give it at level (_median_rank): only paired ratios carry from one machine to another, and one-sided runs of an unchanged call moved -0.6% to +0.6% (the kernel challenge’s measurements, row 3616). |
device_time | A session of GPU device timing (DeviceTime): CUPTI subscribed for each call it reads and detached from the process when it ends, however it ends, so a profile capture or Nsight can subscribe after. |
eviction | The buffer an eviction reads: EVICTION_BYTES of zeros, on the default device. |
samples | Seconds per call of fn(*args), one sample a synchronized group of calls calls, after warmup groups. |
EVICTION_BYTES
Section titled “EVICTION_BYTES”EVICTION_BYTES = 256 << 20The bytes an eviction reads (eviction, behind): over twice the L2 of every GPU Dew targets (an
H100’s 50 MB, a B200’s 126 MB), so a read of them leaves none of a timed call’s operands in it.
Benchmark
Section titled “Benchmark”class Benchmark( seconds: Mapping[Hashable, tuple[float, ...]], outputs: Mapping[Hashable, T],)What benchmark measured: each call’s timed runs in seconds, in order, and its last output.
table() and a notebook’s display show each call’s median. Labels that are all pairs,
(row, column), lay out as a grid of rows by columns, a sampler against its step counts say.
Benchmark.median
Section titled “Benchmark.median”def median(label: Hashable) -> floatBenchmark.table
Section titled “Benchmark.table”def table() -> strThe medians as aligned plain text, one row a label (or a row of pair labels).
Comparison
Section titled “Comparison”class Comparison( ratios: tuple[float, ...], ratio: float, interval: tuple[float, float], alone: tuple[float, float],)A call’s paired ratios to its reference judged (judged): each pair’s ratio of the call’s seconds to
the reference’s, the median ratio and its interval (_median_rank), and the reference against itself
(its A/A), whose interval is the placement’s own noise: a difference is one where interval lies apart
from 1 and outside alone (differs).
differs: bool-
Whether the call’s time differs from its reference’s: its interval apart from 1 and outside the A/A’s, so neither chance nor the placement’s own noise accounts for it.
CuptiHeld
Section titled “CuptiHeld”class CuptiHeld(RuntimeError)CUPTI’s one subscription is another subscriber’s, a profile capture’s or Nsight’s, so no device
time is read (device_time): the kernel tuner keeps a launch’s derived config until a later compile,
and a benchmark reports it.
DeviceTime
Section titled “DeviceTime”class DeviceTime(cupti)The device time of each GPU call device_time’s session reads: its own kernels’ as CUPTI times
them, from their start to their end on the device, which no host dispatch or other process’s work
enters. The caller composes the operands each call reads and the statistic it reports.
CUPTI reads a call after running it once untimed (Mosaic GPU’s profiler warms what it times), so a
call is timed warm; one timed out of the cache runs behind an eviction in one program (behind).
DeviceTime.kernels
Section titled “DeviceTime.kernels”def kernels(fn: Callable, *args=(), evicted: bool = False) -> tuple[KernelTime, ...]Each kernel one call of fn(*args) launched, in launch order, with its device milliseconds,
once its output is ready. A call that launched none is refused (NoKernels): its time would read
as zero. evicted runs the call behind the session’s eviction in one program (behind), its
kernels left out, so its operands are read from memory, not the cache.
DeviceTime.behind
Section titled “DeviceTime.behind”def behind(program: Callable, evicting: jax.Array, *args=()) -> tuple[KernelTime, ...]The kernels of one call of program, a call run behind a read of evicting (behind,
compiled by its caller: the kernel tuner compiles one a config), but the read’s own, which a read
alone names once. A call none of whose own CUPTI read is refused (NoKernels).
DeviceTime.seconds
Section titled “DeviceTime.seconds”def seconds(fn: Callable, *args=(), evicted: bool = False) -> floatOne call’s device seconds, its kernels’ summed (kernels).
KernelTime
Section titled “KernelTime”class KernelTime(NamedTuple)One kernel a call launched and its device time, as CUPTI read it (DeviceTime.kernels).
NoKernels
Section titled “NoKernels”class NoKernels(ValueError)A call CUPTI read none of its own kernels of (DeviceTime), whose time would read as zero.
behind
Section titled “behind”def behind(call: Callable) -> CallableThe program that runs call(*args) behind an eviction, a read of evicting (eviction): called
as (evicting, args), the call waits for the read (an optimization barrier) and both outputs are
returned, so the compiler keeps it. A timing that needs its operands out of the cache times this
program and leaves the read’s kernels out (DeviceTime.behind): CUPTI reads a call after an untimed
run of it, which refills a cache an eviction run before the call emptied.
benchmark
Section titled “benchmark”def benchmark( calls: Mapping[Hashable, Callable[[], T]], *, warmup: int = 1, repeats: int = 3,) -> Benchmark[T]Time each of calls, in order, repeats times after warmup untimed runs (samples), and
return the runs’ seconds and each call’s last output (Benchmark); no two of a call’s outputs
are held at once. Raises ValueError for no timed run or a negative warm-up.
carried
Section titled “carried”def carried( step: Callable[[S], S], state: S, count: int, *, each: bool = False,) -> tuple[S, list[float]]count calls of step, each handed the state the last returned (a training step’s state, a
decoder’s cache), and the seconds they took, with the state the last call returned. With each,
one sample a call, synchronized before the next starts, which times every call alone; otherwise
one sample the window, its calls dispatched as a run dispatches them and synchronized once at its
end, which times their throughput. A carried state is never released between calls, so its
windows are timed apart from samples, whose calls are independent. Refuses no call.
compared
Section titled “compared”def compared( first: Callable[[], T], second: Callable[[], T], *, pairs: int = 400, warmup: int = 3, level: float = 0.95,) -> Comparisonfirst against second, each timed as samples times one call, in pairs interleaved pairs whose
order a fair coin picks each pair (a call that runs second in a pair runs on whatever the first left),
and second against itself in as many pairs, read as the median of each pair’s ratio of seconds and
the interval its pairs’ order statistics give it at level (_median_rank): only paired ratios carry
from one machine to another, and one-sided runs of an unchanged call moved -0.6% to +0.6% (the kernel
challenge’s measurements, row 3616). The coin makes the A/A’s ratios fall above and below 1 as
independent fair coins, however the timings depend on one another, so its interval holds 1 with chance
at least level, exactly; the pair’s holds its median ratio so where its pairs are independent. The
caller binds each call to fresh inputs where its time depends on its data (a MoE’s routing). Raises
ValueError where pairs reach no interval at level.
device_time
Section titled “device_time”def device_time() -> Iterator[DeviceTime]A session of GPU device timing (DeviceTime): CUPTI subscribed for each call it reads and
detached from the process when it ends, however it ends, so a profile capture or Nsight can
subscribe after. CUPTI takes one subscriber; where another holds it, a read raises CuptiHeld.
Dew’s one reader of CUPTI (Mosaic GPU’s profiler’s Cupti).
eviction
Section titled “eviction”def eviction() -> jax.ArrayThe buffer an eviction reads: EVICTION_BYTES of zeros, on the default device.
samples
Section titled “samples”def samples( fn: Callable[..., T], *args=(), warmup: int = 3, repeats: int = 10, calls: int = 1,) -> list[float]Seconds per call of fn(*args), one sample a synchronized group of
calls calls, after warmup groups. With one call a group each sample is
a call’s latency, and its result is released before the next call starts,
so no two calls’ outputs are ever held at once; more calls a group queue
their dispatches and synchronize once, which measures throughput. The
caller takes the statistic and units it reports.