Skip to content

dew.telemetry.benchmark

Wall-clock timing of calls: samples, which every benchmark tool in Dew measures independent calls with, carried, which times calls that carry a state from one to the next, benchmark, which lays out many calls’ timings as a table, and compared, which reads two calls against each other by their paired ratio; a GPU call’s own kernels’ device time, as CUPTI reads it (device_time), which the kernel tuner and the kernel benchmarks read; and the process’s resident memory as the kernel reports it (resident_memory), which the memory benchmarks read.

A run is timed from the call to the moment its output is ready on the devices (jax.block_until_ready), so an asynchronous dispatch is timed as the work it starts, and reading the output back to the host is not timed. The warm-up runs, which compile what the call traces, are not kept. The caller passes what is timed, a jitted function bound to its inputs: nothing is jitted here.

NameSummary
EVICTION_BYTESThe bytes an eviction reads (eviction, behind): over twice the L2 of every GPU Dew targets (an H100’s 50 MB, a B200’s 126 MB), so a read of them leaves none of a timed call’s operands in it.
BenchmarkWhat benchmark measured: each call’s timed runs in seconds, in order, and its last output.
ComparisonA call’s paired ratios to its reference judged (judged): each pair’s ratio of the call’s seconds to the reference’s, the median ratio and its interval (_median_rank), and the reference against itself (its A/A), whose interval is the placement’s own noise: a difference is one where interval lies apart from 1 and outside alone (differs).
CuptiHeldCUPTI’s one subscription is another subscriber’s, a profile capture’s or Nsight’s, so no device time is read (device_time): the kernel tuner keeps a launch’s derived config until a later compile, and a benchmark reports it.
DeviceTimeThe device time of each GPU call device_time’s session reads: its own kernels’ as CUPTI times them, from their start to their end on the device, which no host dispatch or other process’s work enters.
KernelTimeOne kernel a call launched and its device time, as CUPTI read it (DeviceTime.kernels).
NoKernelsA call CUPTI read none of its own kernels of (DeviceTime), whose time would read as zero.
behindThe program that runs call(*args) behind an eviction, a read of evicting (eviction): called as (evicting, args), the call waits for the read (an optimization barrier) and both outputs are returned, so the compiler keeps it.
benchmarkTime each of calls, in order, repeats times after warmup untimed runs (samples), and return the runs’ seconds and each call’s last output (Benchmark); no two of a call’s outputs are held at once.
carriedcount calls of step, each handed the state the last returned (a training step’s state, a decoder’s cache), and the seconds they took, with the state the last call returned.
comparedfirst against second, each timed as samples times one call, in pairs interleaved pairs whose order a fair coin picks each pair (a call that runs second in a pair runs on whatever the first left), and second against itself in as many pairs, read as the median of each pair’s ratio of seconds and the interval its pairs’ order statistics give it at level (_median_rank): only paired ratios carry from one machine to another, and one-sided runs of an unchanged call moved -0.6% to +0.6% (the kernel challenge’s measurements, row 3616).
device_timeA session of GPU device timing (DeviceTime): CUPTI subscribed for each call it reads and detached from the process when it ends, however it ends, so a profile capture or Nsight can subscribe after.
evictionThe buffer an eviction reads: EVICTION_BYTES of zeros, on the default device.
samplesSeconds per call of fn(*args), one sample a synchronized group of calls calls, after warmup groups.

attribute source

EVICTION_BYTES = 256 << 20

The bytes an eviction reads (eviction, behind): over twice the L2 of every GPU Dew targets (an H100’s 50 MB, a B200’s 126 MB), so a read of them leaves none of a timed call’s operands in it.

dataclass source

class Benchmark(
seconds: Mapping[Hashable, tuple[float, ...]],
outputs: Mapping[Hashable, T],
)

What benchmark measured: each call’s timed runs in seconds, in order, and its last output.

table() and a notebook’s display show each call’s median. Labels that are all pairs, (row, column), lay out as a grid of rows by columns, a sampler against its step counts say.

def median(label: Hashable) -> float
def table() -> str

The medians as aligned plain text, one row a label (or a row of pair labels).

dataclass source

class Comparison(
ratios: tuple[float, ...],
ratio: float,
interval: tuple[float, float],
alone: tuple[float, float],
)

A call’s paired ratios to its reference judged (judged): each pair’s ratio of the call’s seconds to the reference’s, the median ratio and its interval (_median_rank), and the reference against itself (its A/A), whose interval is the placement’s own noise: a difference is one where interval lies apart from 1 and outside alone (differs).

differs: bool

Whether the call’s time differs from its reference’s: its interval apart from 1 and outside the A/A’s, so neither chance nor the placement’s own noise accounts for it.

class source

class CuptiHeld(RuntimeError)

CUPTI’s one subscription is another subscriber’s, a profile capture’s or Nsight’s, so no device time is read (device_time): the kernel tuner keeps a launch’s derived config until a later compile, and a benchmark reports it.

class source

class DeviceTime(cupti)

The device time of each GPU call device_time’s session reads: its own kernels’ as CUPTI times them, from their start to their end on the device, which no host dispatch or other process’s work enters. The caller composes the operands each call reads and the statistic it reports.

CUPTI reads a call after running it once untimed (Mosaic GPU’s profiler warms what it times), so a call is timed warm; one timed out of the cache runs behind an eviction in one program (behind).

def kernels(fn: Callable, *args=(), evicted: bool = False) -> tuple[KernelTime, ...]

Each kernel one call of fn(*args) launched, in launch order, with its device milliseconds, once its output is ready. A call that launched none is refused (NoKernels): its time would read as zero. evicted runs the call behind the session’s eviction in one program (behind), its kernels left out, so its operands are read from memory, not the cache.

def behind(program: Callable, evicting: jax.Array, *args=()) -> tuple[KernelTime, ...]

The kernels of one call of program, a call run behind a read of evicting (behind, compiled by its caller: the kernel tuner compiles one a config), but the read’s own, which a read alone names once. A call none of whose own CUPTI read is refused (NoKernels).

def seconds(fn: Callable, *args=(), evicted: bool = False) -> float

One call’s device seconds, its kernels’ summed (kernels).

class source

class KernelTime(NamedTuple)

One kernel a call launched and its device time, as CUPTI read it (DeviceTime.kernels).

class source

class NoKernels(ValueError)

A call CUPTI read none of its own kernels of (DeviceTime), whose time would read as zero.

function source

def behind(call: Callable) -> Callable

The program that runs call(*args) behind an eviction, a read of evicting (eviction): called as (evicting, args), the call waits for the read (an optimization barrier) and both outputs are returned, so the compiler keeps it. A timing that needs its operands out of the cache times this program and leaves the read’s kernels out (DeviceTime.behind): CUPTI reads a call after an untimed run of it, which refills a cache an eviction run before the call emptied.

function source

def benchmark(
calls: Mapping[Hashable, Callable[[], T]],
*,
warmup: int = 1,
repeats: int = 3,
) -> Benchmark[T]

Time each of calls, in order, repeats times after warmup untimed runs (samples), and return the runs’ seconds and each call’s last output (Benchmark); no two of a call’s outputs are held at once. Raises ValueError for no timed run or a negative warm-up.

function source

def carried(
step: Callable[[S], S],
state: S,
count: int,
*,
each: bool = False,
) -> tuple[S, list[float]]

count calls of step, each handed the state the last returned (a training step’s state, a decoder’s cache), and the seconds they took, with the state the last call returned. With each, one sample a call, synchronized before the next starts, which times every call alone; otherwise one sample the window, its calls dispatched as a run dispatches them and synchronized once at its end, which times their throughput. A carried state is never released between calls, so its windows are timed apart from samples, whose calls are independent. Refuses no call.

function source

def compared(
first: Callable[[], T],
second: Callable[[], T],
*,
pairs: int = 400,
warmup: int = 3,
level: float = 0.95,
) -> Comparison

first against second, each timed as samples times one call, in pairs interleaved pairs whose order a fair coin picks each pair (a call that runs second in a pair runs on whatever the first left), and second against itself in as many pairs, read as the median of each pair’s ratio of seconds and the interval its pairs’ order statistics give it at level (_median_rank): only paired ratios carry from one machine to another, and one-sided runs of an unchanged call moved -0.6% to +0.6% (the kernel challenge’s measurements, row 3616). The coin makes the A/A’s ratios fall above and below 1 as independent fair coins, however the timings depend on one another, so its interval holds 1 with chance at least level, exactly; the pair’s holds its median ratio so where its pairs are independent. The caller binds each call to fresh inputs where its time depends on its data (a MoE’s routing). Raises ValueError where pairs reach no interval at level.

function source

def device_time() -> Iterator[DeviceTime]

A session of GPU device timing (DeviceTime): CUPTI subscribed for each call it reads and detached from the process when it ends, however it ends, so a profile capture or Nsight can subscribe after. CUPTI takes one subscriber; where another holds it, a read raises CuptiHeld. Dew’s one reader of CUPTI (Mosaic GPU’s profiler’s Cupti).

function source

def eviction() -> jax.Array

The buffer an eviction reads: EVICTION_BYTES of zeros, on the default device.

function source

def samples(
fn: Callable[..., T],
*args=(),
warmup: int = 3,
repeats: int = 10,
calls: int = 1,
) -> list[float]

Seconds per call of fn(*args), one sample a synchronized group of calls calls, after warmup groups. With one call a group each sample is a call’s latency, and its result is released before the next call starts, so no two calls’ outputs are ever held at once; more calls a group queue their dispatches and synchronize once, which measures throughput. The caller takes the statistic and units it reports.