Skip to content

dew.nn.kernels

Pallas kernels for the operations XLA does not fuse.

Each module here implements one operation and exports the predicate that says which shapes and backends its kernel covers. The XLA form that the kernel replaces stays in the module the operation belongs to. That form is the reference the kernel’s tests compare against, and the path every other shape and backend takes.

NameSummary
PALLAS_CHOICESThe kernel names that run Dew’s Pallas or Triton kernels, which lower through JAX’s Pallas Triton backend for a GPU only: what the CPU dry run lowers (dew.training._lowering) and the kernel lab verifies on the GPU.
TARGETS
GpuTargetOne generation’s device prefixes, dense peaks, block memory and measured kernel row.
assumedWithin the block, choose kernels as on a device of generation (‘sm80’), whatever device this process has: device_generation, bf16_dot_runs and triton_runs answer for it, so every kernel choice and refusal acts as on that GPU while a step traces and lowers for one on a CPU (dew.training._lowering).
backendThe platform a call’s code is chosen for: placed_platform(), or within assumed the assumed generation’s, ‘gpu’ for an smXY and ‘tpu’ for a TPU’s, so a choice keyed on the platform acts as on that device while a step traces for it on another.
bf16_dot_runsReturn whether the default device multiplies bf16 operands into an fp32 sum as one dot algorithm.
chosen_kernelThe kernel chosen names for choice within its block, else default: a choice a site decides without a selector (the attention template’s form, a step’s narrow copies).
device_generationReturn the hardware generation of the devices a call is placed on.
gpu_meshmesh’s axes over abstract devices of the CUDA generation (‘sm80’), none of which this process needs: a program traced on it, under assumed(generation), lowers its Pallas kernels to Triton, and its Mosaic GPU kernels to that GPU’s instructions, for that GPU on any host (TrainerConfig.lower_for; the kernel benchmark’s oracle; tests/kernel_compile.py).
gpu_targetThe target for a generation, reported device kind or case-insensitive lab alias.
ssd_chunk_scan
ssd_kernel_platformReturn the backend to build this dtype scan’s kernel for, or None for the XLA path.
ssd_kernel_runsReturn whether the SSD kernel is chosen for this geometry and the scan’s dtype.
triton_runsReturn whether Dew’s Pallas GPU (Triton) kernels can run here.

attribute source

PALLAS_CHOICES = frozenset({'pallas', 'fla'})

The kernel names that run Dew’s Pallas or Triton kernels, which lower through JAX’s Pallas Triton backend for a GPU only: what the CPU dry run lowers (dew.training._lowering) and the kernel lab verifies on the GPU. ‘flash’ and ‘cudnn’ name CUDA libraries.

attribute source

TARGETS

dataclass source

class GpuTarget(
generation: str,
device_kind: str,
shared_limit: int | None = None,
dense_peaks: Mapping[str, Mapping[str, float]] = dict(),
aliases: tuple[str, ...] = (),
multiprocessors: int | None = None,
clock_hz: float | None = None,
memory_bandwidth: float | None = None,
measured: Mapping[str, Measured] = dict(),
int8_triton_gemm: bool = False,
triton_gemm_fusions: bool = True,
late_weight_gradients: bool = False,
step_seconds: float | None = None,
kernel_step_seconds: float | None = None,
)

One generation’s device prefixes, dense peaks, block memory and measured kernel row.

dense_peaks maps reported device-kind prefixes to precision rates, so PCIe/SXM and GeForce/data-center variants retain their own peaks. device_kind is the canonical device for a generation, or the reported SKU a reader asked for. Empty resources remain unknown. The standard-library-only table can also be read by a Torch reference’s environment without importing JAX.

multiprocessors is the canonical device’s streaming multiprocessors, a real device’s own where it reports them (gpu_target), which a launch sizes its programs by (dew.nn.kernels.decode_attention). clock_hz and memory_bandwidth are the canonical device’s boost clock and device-memory bytes per second, a real device’s own likewise; measured maps device-kind prefixes to the facts a part was measured to (part): its stream, which prices its device bytes (memory_rate), and its launches’ and hand-offs’ fixed seconds (launch_seconds, handoff_seconds). Every other fact a kernel’s cost model reads (dew.nn.kernels.templates.cost) is its instruction family’s (architecture): its tensor rates, warpgroup MMA and tensor memory. A row states a device’s datasheet, its peaks, which MFU reads, and what a compile-only target of it needs, beside the few facts measured on the part itself. This table and architecture are the one place a choice is keyed on the hardware; any other code reads their facts, a library’s support rule the compute capability (capability), and a launch config its launch’s rule from those facts (dew.nn.kernels.tuning), as tools/lint_slop.py’s SLOP016 holds.

int8_triton_gemm: bool

Whether XLA’s Triton GEMM fusion computes int8 dots right on this GPU: on none measured, an A100 and an RTX PRO 6000 Blackwell, at jaxlib 0.11.2 (openxla/xla#51016; dew.nn.kernels.options.compiler_options).

triton_gemm_fusions: bool

Whether a training step keeps XLA’s Triton GEMM fusions on this GPU (dew.training.memory): off where a step measured cuBLAS’s GEMMs faster with them off, an A100’s (26a01bc18), an L4’s (71966bcea) and an H100’s (0fb7f065e), unless a module the step runs keeps them (TritonGemm).

late_weight_gradients: bool

Whether XLA’s schedule of a training step for this GPU runs its weight gradients after the whole backward where the step’s head takes its gradient a tile at a time, unfused, holding the products’ operands to then, as the derivation runs them (dew.training.footprint.sunk): an H100’s every one, a B200’s and a B300’s some, which running every one there bounds from above (row 3341’s calibration, compile-only, TARGETS’ comments).

step_seconds: float | None

One dependent step’s latency on XLA’s path: an iteration of a loop XLA lowers, its launches and its predicate, each reading the last (the work-span model’s step, templates.cost.Work.span), measured per GPU (FS-STEP-SECONDS1’s receipts); None where none was, so a form’s span there is priced by no time.

kernel_step_seconds: float | None

One dependent step’s latency within a kernel: a tile’s 128-lane reduction broadcast into the next step’s operand, in one program, each reading the last (a template walk’s span, templates.cost.walked), measured per GPU as the slope over 1,024 to 16,384 steps (FS-STEP-SECONDS1’s kernel path); None where none was.

architecture: Architecture

The target’s instruction family’s facts (architecture), none where it belongs to none.

wgmma: bool

Whether the GPU’s matmuls run Hopper’s warpgroup MMA, which Pallas Triton pipelines by holding every stage of a tile in shared memory (dew.nn.kernels.grouped_matmul).

tmem_columns: int | None

A Blackwell multiprocessor’s tensor memory columns, the bound a kernel’s tensor memory takes (tests/kernel_compile.py); None on a GPU without it.

scoped_vmem: int | None

A TPU’s scoped VMEM limit in bytes at XLA’s default, past which a Mosaic kernel refuses to compile (dew.nn.kernels.splash.block_sizes); None where no rule states it.

vmem_bytes: int | None

A TPU’s VMEM capacity in bytes, the most a Mosaic kernel may raise its scoped limit to (vmem_limit_bytes); None on GPUs and where no rule states it.

multiprocessor: Multiprocessor

The resources of this target’s CUDA multiprocessor (Architecture.multiprocessor); a TPU has none, which no CUDA launch’s derivation reads.

part: Measured | None

The facts this device’s part was measured to (measured); None for a part no one measured.

memory_rate: float | None

Device-memory bytes a second a long transfer streams at: the part’s measured plateau (part), else its clocks’ bandwidth (memory_bandwidth).

fixed_seconds: float | None

The seconds a kernel’s stream spends before its bytes arrive, its launch and first byte, measured on the part (Measured.fixed); None where no one measured it.

launch_seconds: float | None

The seconds a kernel launch adds to a program of kernels one after another, measured on the part (Measured.launch); None where no one measured it.

handoff_seconds: float | None

A GPU-scope semaphore hand-off’s seconds between two programs of one launch, measured on the part (Measured.handoff); None where no one measured it.

def tensor_rate(mode: str) -> float | None

Dense tensor-core FLOPs per clock per multiprocessor in arithmetic mode, or None where the generation states none: the generation’s, lowered where the SKU’s catalogued dense bf16 peak is below what its multiprocessors and clock give at that rate, as a GeForce part’s half-rate fp32 accumulation reads; IEEE fp32 on the CUDA cores keeps its rate. A part reporting fewer multiprocessors than its catalogue counts (a MIG slice) keeps the generation’s rate: the catalogue never raises it.

function source

def assumed(generation: str | GpuTarget | None) -> Iterator[None]

Within the block, choose kernels as on a device of generation (‘sm80’), whatever device this process has: device_generation, bf16_dot_runs and triton_runs answer for it, so every kernel choice and refusal acts as on that GPU while a step traces and lowers for one on a CPU (dew.training._lowering). Given a target’s facts (a GPU no row names, its multiprocessors, shared memory, clock and bandwidth its own), gpu_target answers with them, so every launch is sized and costed by them alone. None assumes nothing. JAX keys its trace and compile caches on the device assumed (dew._cache.KERNEL_ASSUMED), so a function jitted for one traces again for another.

function source

def backend() -> str

The platform a call’s code is chosen for: placed_platform(), or within assumed the assumed generation’s, ‘gpu’ for an smXY and ‘tpu’ for a TPU’s, so a choice keyed on the platform acts as on that device while a step traces for it on another.

function source

def bf16_dot_runs() -> bool

Return whether the default device multiplies bf16 operands into an fp32 sum as one dot algorithm.

That algorithm is BF16_BF16_F32. Every TPU and CPU has it, and so does a GPU from generation BF16_GPU on.

function source

def chosen_kernel(choice: str, default: str) -> str

The kernel chosen names for choice within its block, else default: a choice a site decides without a selector (the attention template’s form, a step’s narrow copies).

function source

def device_generation() -> str

Return the hardware generation of the devices a call is placed on.

The generation is ‘sm89’ for a GPU of compute capability 8.9, ‘v6e’ for a TPU v6e, and the platform’s name otherwise; within assumed, the one it names. A call placed off the process’s default backend (placed_platform), as a host-layout run’s on a GPU host’s CPU, takes its platform’s name, so every choice keyed on the target follows it there.

function source

def gpu_mesh(mesh: Mesh, generation: str) -> AbstractMesh

mesh’s axes over abstract devices of the CUDA generation (‘sm80’), none of which this process needs: a program traced on it, under assumed(generation), lowers its Pallas kernels to Triton, and its Mosaic GPU kernels to that GPU’s instructions, for that GPU on any host (TrainerConfig.lower_for; the kernel benchmark’s oracle; tests/kernel_compile.py). Its devices state their compute capability, which Mosaic GPU lowers for (its utils._infer_arch; a device that states none it takes for sm90’s, and an sm80 kernel’s cp.async waits do not lower), but an sm90’s, which Mosaic’s default already is: stated, its 9.0 sends a warpgroup-MMA kernel’s ptxas a lower default SM than the sm_90a its PTX targets (‘SM version specified by .target is higher than default SM version assumed’, FAM-contract3). JAX 0.12’s abstract device states it (AbstractDevice.compute_capability).

function source

def gpu_target(identifier: str | None = None) -> GpuTarget

The target for a generation, reported device kind or case-insensitive lab alias.

None is this process’s first device’s (or the facts assumed was given): its generation is device_generation’s, the one seam that assumed and tests set, and its device kind only picks the SKU’s peaks within that generation; a real GPU the table names no SKU of keeps its own kind, so its peaks stay unknown, while an assumed or patched generation takes the generation’s canonical device. SKU queries retain that SKU’s peaks. Unknown kinds have empty resources.

attribute source

ssd_chunk_scan = derivatives.contracted('test_the_tpu_ssd_scan_holds_its_orders')(derivatives.declared_derivative(derivatives.KernelGlue(), _ssd_scan, _ssd_fwd, _ssd_bwd, nondiff_argnums=(5, 6)))

function source

def ssd_kernel_platform(
chunk_size: int,
head_dim: int,
state_size: int,
*,
dtype: DTypeLike,
) -> str | None

Return the backend to build this dtype scan’s kernel for, or None for the XLA path.

The scan’s route is the selector’s to report (recurrence_chunk.SCAN, generation’s Route collection), where the kernel report reads it (row 971).

function source

def ssd_kernel_runs(
chunk_size: int,
head_dim: int,
state_size: int,
backend: str,
*,
dtype: DTypeLike,
) -> bool

Return whether the SSD kernel is chosen for this geometry and the scan’s dtype.

The kernel needs a tpu backend, a float32 scan (a wider scan takes the XLA path in its own dtype), a chunk long enough to pay for a program, three power-of-two widths so that Mosaic’s tiling wastes no lanes, and a tile inside the per-program budget. This kernel’s one-program form ran 6x to 12x slower than XLA as Triton on an RTX 4080 and overflowed shared memory at chunks of 128 and 256 (docs/performance.md); a GPU runs mamba_ssm’s decomposition into fourteen kernels (dew.nn.kernels.mamba_ssd).

function source

def triton_runs() -> bool

Return whether Dew’s Pallas GPU (Triton) kernels can run here.

This is the one eligibility rule for those kernels: a GPU of compute capability 8.0 or later, the same bound JAX’s own Pallas lowerings apply. A T4 fails to compile them (“Triton support is only enabled for cc>=8.0”). Within assumed, the GPU is the assumed generation’s.