dew.nn.kernels
Pallas kernels for the operations XLA does not fuse.
Each module here implements one operation and exports the predicate that says which shapes and backends its kernel covers. The XLA form that the kernel replaces stays in the module the operation belongs to. That form is the reference the kernel’s tests compare against, and the path every other shape and backend takes.
| Name | Summary |
|---|---|
PALLAS_CHOICES | The kernel names that run Dew’s Pallas or Triton kernels, which lower through JAX’s Pallas Triton backend for a GPU only: what the CPU dry run lowers (dew.training._lowering) and the kernel lab verifies on the GPU. |
TARGETS | |
GpuTarget | One generation’s device prefixes, dense peaks, block memory and measured kernel row. |
assumed | Within the block, choose kernels as on a device of generation (‘sm80’), whatever device this process has: device_generation, bf16_dot_runs and triton_runs answer for it, so every kernel choice and refusal acts as on that GPU while a step traces and lowers for one on a CPU (dew.training._lowering). |
backend | The platform a call’s code is chosen for: placed_platform(), or within assumed the assumed generation’s, ‘gpu’ for an smXY and ‘tpu’ for a TPU’s, so a choice keyed on the platform acts as on that device while a step traces for it on another. |
bf16_dot_runs | Return whether the default device multiplies bf16 operands into an fp32 sum as one dot algorithm. |
chosen_kernel | The kernel chosen names for choice within its block, else default: a choice a site decides without a selector (the attention template’s form, a step’s narrow copies). |
device_generation | Return the hardware generation of the devices a call is placed on. |
gpu_mesh | mesh’s axes over abstract devices of the CUDA generation (‘sm80’), none of which this process needs: a program traced on it, under assumed(generation), lowers its Pallas kernels to Triton, and its Mosaic GPU kernels to that GPU’s instructions, for that GPU on any host (TrainerConfig.lower_for; the kernel benchmark’s oracle; tests/kernel_compile.py). |
gpu_target | The target for a generation, reported device kind or case-insensitive lab alias. |
ssd_chunk_scan | |
ssd_kernel_platform | Return the backend to build this dtype scan’s kernel for, or None for the XLA path. |
ssd_kernel_runs | Return whether the SSD kernel is chosen for this geometry and the scan’s dtype. |
triton_runs | Return whether Dew’s Pallas GPU (Triton) kernels can run here. |
PALLAS_CHOICES
Section titled “PALLAS_CHOICES”PALLAS_CHOICES = frozenset({'pallas', 'fla'})The kernel names that run Dew’s Pallas or Triton kernels, which lower through JAX’s Pallas
Triton backend for a GPU only: what the CPU dry run lowers (dew.training._lowering) and the kernel
lab verifies on the GPU. ‘flash’ and ‘cudnn’ name CUDA libraries.
TARGETS
Section titled “TARGETS”TARGETSGpuTarget
Section titled “GpuTarget”class GpuTarget( generation: str, device_kind: str, shared_limit: int | None = None, dense_peaks: Mapping[str, Mapping[str, float]] = dict(), aliases: tuple[str, ...] = (), multiprocessors: int | None = None, clock_hz: float | None = None, memory_bandwidth: float | None = None, measured: Mapping[str, Measured] = dict(), int8_triton_gemm: bool = False, triton_gemm_fusions: bool = True, late_weight_gradients: bool = False, step_seconds: float | None = None, kernel_step_seconds: float | None = None,)One generation’s device prefixes, dense peaks, block memory and measured kernel row.
dense_peaks maps reported device-kind prefixes to precision rates, so
PCIe/SXM and GeForce/data-center variants retain their own peaks. device_kind
is the canonical device for a generation, or the reported SKU a reader asked
for. Empty resources remain unknown. The standard-library-only table can
also be read by a Torch reference’s environment without importing JAX.
multiprocessors is the canonical device’s streaming multiprocessors, a real
device’s own where it reports them (gpu_target), which a launch sizes its
programs by (dew.nn.kernels.decode_attention). clock_hz and
memory_bandwidth are the canonical device’s boost clock and device-memory
bytes per second, a real device’s own likewise; measured maps device-kind
prefixes to the facts a part was measured to (part): its stream, which
prices its device bytes (memory_rate), and its launches’ and hand-offs’
fixed seconds (launch_seconds, handoff_seconds). Every other fact a kernel’s
cost model reads (dew.nn.kernels.templates.cost) is its instruction
family’s (architecture): its tensor rates, warpgroup MMA and tensor memory. A row
states a device’s datasheet, its peaks, which MFU reads, and what a compile-only
target of it needs, beside the few facts measured on the part itself. This
table and architecture are the one
place a choice is keyed on the hardware; any other code reads their facts, a
library’s support rule the compute capability (capability), and a launch
config its launch’s rule from those facts (dew.nn.kernels.tuning), as
tools/lint_slop.py’s SLOP016 holds.
int8_triton_gemm: bool-
Whether XLA’s Triton GEMM fusion computes int8 dots right on this GPU: on none measured, an A100 and an RTX PRO 6000 Blackwell, at jaxlib 0.11.2 (openxla/xla#51016;
dew.nn.kernels.options.compiler_options). triton_gemm_fusions: bool-
Whether a training step keeps XLA’s Triton GEMM fusions on this GPU (
dew.training.memory): off where a step measured cuBLAS’s GEMMs faster with them off, an A100’s (26a01bc18), an L4’s (71966bcea) and an H100’s (0fb7f065e), unless a module the step runs keeps them (TritonGemm). late_weight_gradients: bool-
Whether XLA’s schedule of a training step for this GPU runs its weight gradients after the whole backward where the step’s head takes its gradient a tile at a time, unfused, holding the products’ operands to then, as the derivation runs them (
dew.training.footprint.sunk): an H100’s every one, a B200’s and a B300’s some, which running every one there bounds from above (row 3341’s calibration, compile-only,TARGETS’ comments). step_seconds: float | None-
One dependent step’s latency on XLA’s path: an iteration of a loop XLA lowers, its launches and its predicate, each reading the last (the work-span model’s step,
templates.cost.Work.span), measured per GPU (FS-STEP-SECONDS1’s receipts); None where none was, so a form’s span there is priced by no time. kernel_step_seconds: float | None-
One dependent step’s latency within a kernel: a tile’s 128-lane reduction broadcast into the next step’s operand, in one program, each reading the last (a template walk’s span,
templates.cost.walked), measured per GPU as the slope over 1,024 to 16,384 steps (FS-STEP-SECONDS1’s kernel path); None where none was. architecture: Architecture-
The target’s instruction family’s facts (
architecture), none where it belongs to none. wgmma: bool-
Whether the GPU’s matmuls run Hopper’s warpgroup MMA, which Pallas Triton pipelines by holding every stage of a tile in shared memory (
dew.nn.kernels.grouped_matmul). tmem_columns: int | None-
A Blackwell multiprocessor’s tensor memory columns, the bound a kernel’s tensor memory takes (tests/kernel_compile.py); None on a GPU without it.
scoped_vmem: int | None-
A TPU’s scoped VMEM limit in bytes at XLA’s default, past which a Mosaic kernel refuses to compile (
dew.nn.kernels.splash.block_sizes); None where no rule states it. vmem_bytes: int | None-
A TPU’s VMEM capacity in bytes, the most a Mosaic kernel may raise its scoped limit to (
vmem_limit_bytes); None on GPUs and where no rule states it. multiprocessor: Multiprocessor-
The resources of this target’s CUDA multiprocessor (
Architecture.multiprocessor); a TPU has none, which no CUDA launch’s derivation reads. part: Measured | None-
The facts this device’s part was measured to (
measured); None for a part no one measured. memory_rate: float | None-
Device-memory bytes a second a long transfer streams at: the part’s measured plateau (
part), else its clocks’ bandwidth (memory_bandwidth). fixed_seconds: float | None-
The seconds a kernel’s stream spends before its bytes arrive, its launch and first byte, measured on the part (
Measured.fixed); None where no one measured it. launch_seconds: float | None-
The seconds a kernel launch adds to a program of kernels one after another, measured on the part (
Measured.launch); None where no one measured it. handoff_seconds: float | None-
A GPU-scope semaphore hand-off’s seconds between two programs of one launch, measured on the part (
Measured.handoff); None where no one measured it.
GpuTarget.tensor_rate
Section titled “GpuTarget.tensor_rate”def tensor_rate(mode: str) -> float | NoneDense tensor-core FLOPs per clock per multiprocessor in arithmetic mode, or None where
the generation states none: the generation’s, lowered where the SKU’s catalogued dense bf16 peak
is below what its multiprocessors and clock give at that rate, as a GeForce part’s half-rate
fp32 accumulation reads; IEEE fp32 on the CUDA cores keeps its rate. A part reporting fewer
multiprocessors than its catalogue counts (a MIG slice) keeps the generation’s rate: the
catalogue never raises it.
assumed
Section titled “assumed”def assumed(generation: str | GpuTarget | None) -> Iterator[None]Within the block, choose kernels as on a device of generation (‘sm80’), whatever device this
process has: device_generation, bf16_dot_runs and triton_runs answer for it, so every
kernel choice and refusal acts as on that GPU while a step traces and lowers for one on
a CPU (dew.training._lowering). Given a target’s facts (a GPU no row names, its multiprocessors,
shared memory, clock and bandwidth its own), gpu_target answers with them, so every launch is sized
and costed by them alone. None assumes nothing. JAX keys its trace and compile caches on the device
assumed (dew._cache.KERNEL_ASSUMED), so a function jitted for one traces again for another.
backend
Section titled “backend”def backend() -> strThe platform a call’s code is chosen for: placed_platform(), or within assumed the
assumed generation’s, ‘gpu’ for an smXY and ‘tpu’ for a TPU’s, so a choice keyed on the
platform acts as on that device while a step traces for it on another.
bf16_dot_runs
Section titled “bf16_dot_runs”def bf16_dot_runs() -> boolReturn whether the default device multiplies bf16 operands into an fp32 sum as one dot algorithm.
That algorithm is BF16_BF16_F32. Every TPU and CPU has it, and so does a
GPU from generation BF16_GPU on.
chosen_kernel
Section titled “chosen_kernel”def chosen_kernel(choice: str, default: str) -> strThe kernel chosen names for choice within its block, else default: a choice a site decides
without a selector (the attention template’s form, a step’s narrow copies).
device_generation
Section titled “device_generation”def device_generation() -> strReturn the hardware generation of the devices a call is placed on.
The generation is ‘sm89’ for a GPU of compute capability 8.9, ‘v6e’ for a
TPU v6e, and the platform’s name otherwise; within assumed, the one it
names. A call placed off the process’s default backend (placed_platform),
as a host-layout run’s on a GPU host’s CPU, takes its platform’s name, so
every choice keyed on the target follows it there.
gpu_mesh
Section titled “gpu_mesh”def gpu_mesh(mesh: Mesh, generation: str) -> AbstractMeshmesh’s axes over abstract devices of the CUDA generation (‘sm80’), none of which this process
needs: a program traced on it, under assumed(generation), lowers its Pallas kernels to Triton, and
its Mosaic GPU kernels to that GPU’s instructions, for that GPU on any host (TrainerConfig.lower_for;
the kernel benchmark’s oracle; tests/kernel_compile.py). Its devices state their compute capability, which
Mosaic GPU lowers for (its utils._infer_arch; a device that states none it takes for sm90’s, and an
sm80 kernel’s cp.async waits do not lower), but an sm90’s, which Mosaic’s default already is: stated,
its 9.0 sends a warpgroup-MMA kernel’s ptxas a lower default SM than the sm_90a its PTX targets
(‘SM version specified by .target is higher than default SM version assumed’, FAM-contract3). JAX
0.12’s abstract device states it (AbstractDevice.compute_capability).
gpu_target
Section titled “gpu_target”def gpu_target(identifier: str | None = None) -> GpuTargetThe target for a generation, reported device kind or case-insensitive lab alias.
None is this process’s first device’s (or the facts assumed was given): its generation is
device_generation’s, the one seam that assumed and tests set, and its
device kind only picks the SKU’s peaks within that generation; a real GPU the table names no SKU of
keeps its own kind, so its peaks stay unknown, while an assumed or patched
generation takes the generation’s canonical device. SKU queries retain
that SKU’s peaks. Unknown kinds have empty resources.
ssd_chunk_scan
Section titled “ssd_chunk_scan”ssd_chunk_scan = derivatives.contracted('test_the_tpu_ssd_scan_holds_its_orders')(derivatives.declared_derivative(derivatives.KernelGlue(), _ssd_scan, _ssd_fwd, _ssd_bwd, nondiff_argnums=(5, 6)))ssd_kernel_platform
Section titled “ssd_kernel_platform”def ssd_kernel_platform( chunk_size: int, head_dim: int, state_size: int, *, dtype: DTypeLike,) -> str | NoneReturn the backend to build this dtype scan’s kernel for, or None for the XLA path.
The scan’s route is the selector’s to report (recurrence_chunk.SCAN, generation’s Route
collection), where the kernel report reads it (row 971).
ssd_kernel_runs
Section titled “ssd_kernel_runs”def ssd_kernel_runs( chunk_size: int, head_dim: int, state_size: int, backend: str, *, dtype: DTypeLike,) -> boolReturn whether the SSD kernel is chosen for this geometry and the scan’s dtype.
The kernel needs a tpu backend, a float32 scan (a wider scan takes the
XLA path in its own dtype), a chunk long enough to pay for a program,
three power-of-two widths so that Mosaic’s tiling wastes no lanes, and a
tile inside the per-program budget. This kernel’s one-program form ran 6x
to 12x slower than XLA as Triton on an RTX 4080 and overflowed shared
memory at chunks of 128 and 256 (docs/performance.md); a GPU runs
mamba_ssm’s decomposition into fourteen kernels (dew.nn.kernels.mamba_ssd).
triton_runs
Section titled “triton_runs”def triton_runs() -> boolReturn whether Dew’s Pallas GPU (Triton) kernels can run here.
This is the one eligibility rule for those kernels: a GPU of compute
capability 8.0 or later, the same bound JAX’s own Pallas lowerings apply.
A T4 fails to compile them (“Triton support is only enabled for
cc>=8.0”). Within assumed, the GPU is the assumed generation’s.