Skip to content

dew.nn.precision

The precision a bf16 model’s fp32 boundaries multiply at.

A model’s last matmul before a loss (the patch output head, the vocabulary head) keeps an fp32 result, since the loss is fp32; head_product says why the vocabulary head still rounds to bf16 values at the default precision. preferred_element_type=jnp.float32 alone makes that dot’s cotangent fp32, so the mixed f32 x bf16 backward products run at the fp32 rate. XLA’s BF16_BF16_F32 dot algorithm names the arithmetic instead (bf16 operands, fp32 accumulation, whatever the storage), and jax carries precision into the transpose rules, so the backward follows; jax’s own xla attention sets it for this reason (jax-ml/jax#24047). Dew’s products at that algorithm type their operands bf16 instead (bf16_dot_general, which the dot family runs for it): an A100 computes the algorithm wrong where a dot has a vector side or sits below XLA’s GEMM rewrite threshold.

Dew’s test processes run wide CPU arithmetic (WIDE_CPU_ARITHMETIC, which tests/lane_environment.py sets): a CPU then sums every float32 product of the dot family, dot_general, einsum and matmul, in float64 and rounds once (accumulated), since XLA:CPU chooses its fp32 dot accumulation by shape, instruction set and release, and jax 0.12’s summed twice numpy’s error from float64 where 0.11’s summed 0.6 of it. A library process’s products are the plain products on every platform. Convolutions, depthwise ones included, keep XLA’s fp32 sum on a CPU: no convolution’s check 1 failed there, and XLA:CPU’s float64 convolution cost a 3x3 one’s training step 44 times (15.5 to 675 ms on a 4x32x32x128 map), its best rewrite 2 to 4. Should a convolution’s check 1 resolve above its bar, the recorded fix is that rewrite: the kernel’s shifted windows taken inside the float64 scope and contracted with the taps as one 2D product (stride 1 train 3.9 times, stride 2 1.6).

NameSummary
ACCURATEAccurate, scientific work at its cost: float32 throughout, every product that states no precision in IEEE float32 and each op’s accurate form.
DTYPE_PRESETSThe usual dtypes by Lightning’s names: float32 parameters computing in bf16 by default, float32 throughout, bf16 throughout, and float16 compute over float32 parameters.
FP32_SUMSJAX’s dot algorithm of fp32 operands summed in fp32: as a process’s or a scope’s default matmul precision, the statement that its fp32 products sum as an accelerator sums them, which accumulated honors on a CPU.
L2NORM_EPSThe epsilon l2norm adds under its square root: FLA’s, and Dory’s justnorm’s.
MATMUL_PRECISIONSJAX’s matmul precision levels (jax_default_matmul_precision): “default” TF32 for a float32 product on a GPU and one bf16 pass on a TPU, “highest” IEEE float32.
NUMERICS_PRESETSThe named run numerics beside the default (Numerics()).
ROUNDING_STREAMThe stream stochastic rounding draws from: what a training step folds its key with (dew.objectives.base.training_rngs), and what a served row’s rounding key starts from (dew.sampling.text.rounding_keys), apart from dropout’s and sampling’s.
WIDE_CPU_ARITHMETICThe environment variable that, set to 1, has a process’s CPU programs take the arithmetic Dew’s tests hold a CPU to: float32 products summed in float64 and rounded once (accumulated, spread), a loop’s running sum of them compensated so it rounds once too (compensates_loops), and a float32 reduction taken in float64 and rounded once (widened_reduction).
DtypesA run’s dtypes, three as Flax casts them: the parameters’ (its layers’ param_dtype, and the optimizer’s moments’), the compute’s (their dtype) and the outputs’ (the logits and losses a model returns).
NumericsA run’s numerics, set once and recorded with it, which its model, optimizer and trainer read: its dtypes (Dtypes, which a model takes as Flax’s param_dtype and dtype and its optimizer as its moments’; DTYPE_PRESETS names the usual ones by Lightning’s names), the precision every product that states none multiplies at (JAX’s own jax_default_matmul_precision level, or a jax.lax.DotAlgorithmPreset, which a record holds by its name), and each op’s form, its kernels’ fast or accurate arithmetic (dew.nn.kernels.generation.run_precision).
accumulatedproduct(*operands, preferred_element_type=...), a sum of products of operands (a dot, an einsum, a matmul): the plain product, but in a process running wide CPU arithmetic (WIDE_CPU_ARITHMETIC, Dew’s tests), where they multiply in float32 (products_dtype) a CPU sums the products in float64 and rounds the sum once to the dtype the product returns.
activation_and_slopeAn fp32 gate’s act(gate) and its derivative, for ‘silu’ or ‘sigmoid’.
added_torunning (running_sum) with term added: through compensated_add where it is a pair.
asks_default_precisionWhether precision asks for no more than the default.
at_least_fp32The dtype arithmetic Dew keeps from rounding below float32 runs in: float32 for float32, every narrower float and a module’s unset dtype (None), and dtype itself where it is wider.
at_precisionfn traced, forward and backward, at precision, the default included: a product that takes no precision of its own (a Pallas dot, jax.nn’s xla attention) reads it from the configuration at trace time (jax.default_matmul_precision), and a custom derivative’s backward is traced after the forward returns, so it is held under the same setting by hand.
bf16_dot_generaljax.lax.dot_general(lhs, rhs, dimension_numbers) multiplied as XLA’s BF16_BF16_F32 dot algorithm states the arithmetic: the operands rounded to bf16 (to nearest even), each product exact and summed in fp32 to an fp32 result.
bf16_einsumjnp.einsum(subscripts, *operands) in fp32, its every product bf16_dot_general’s: einsum plans the contraction (its path, broadcasts, and each operand’s own sums, diagonals and transposes, which form no product and run in fp32) and hands each pairwise product to bf16_dot_general as its _dot_general.
bf16_logitsfp32 logits rounded to bf16 and held in fp32, as rounded_to rounds them, written as bf16 under CUDA.
bf16_productjnp.einsum(subscripts, hidden, head) multiplied as bf16 and its result rounded to bf16, held in fp32: the one product a bf16 head computes (head_product) and a tiled head computes over each tile, so the two round each logit alike.
biasedproduct plus bias broadcast over its leading axes, in the dtype their addition promotes to, as flax’s layer adds them.
capped_logitsSoftcapped logits over a sampling temperature, which an engine applies to what the model’s head returns.
compensated_addAdd two double-float values, (hi, lo) pairs whose sum is the value: hi is the sum rounded and lo what the rounding dropped (the his’ TwoSum, the los added to its remainder, then the pair’s own TwoSum), so a running sum keeps fp32’s precision squared and its hi is the running value rounded once, where a pair left unrenormalised keeps the plain running sum in hi, every addition’s rounding in it.
compensated_differencelater - earlier of two double-float values, rounded once: the halves’ differences summed, so the difference of two near values (a range of a running sum) keeps its own precision, where the his’ difference alone carries their whole magnitude’s rounding: a one-token range of a decay’s running sum near -78 off by 4e-6.
compensated_sumThe sum of values over axes, rounded once: one XLA reduction whose accumulator is a double-float (hi, lo) pair combined by compensated_add, so it partitions as any reduction does, and what the roundings dropped is added back at the end.
compensated_totalA compensated sum’s value, its (hi, lo) pair rounded once, or the plain sum of the same values where that reaches an infinity: +-inf or NaN as jnp.sum’s is, where every residual past the infinity is inf - inf.
compensates_loopsWhether a running sum of float32 products carried across a loop keeps what each addition drops (compensated_add), so the loop’s sum rounds once as one product’s does: where a CPU sums each product in float64 (accumulated), in a process running Dew’s wide CPU arithmetic (WIDE_CPU_ARITHMETIC).
configurable_precisionWhether jax.default_matmul_precision can name precision: one precision or dot algorithm preset, unset or default included; not a pair of precisions or a dot algorithm.
configured_precisionA context in which a product that takes no precision of its own (jax.nn’s xla attention) multiplies at precision, through jax.default_matmul_precision; an unset or default precision changes nothing, a floor the configured one meets.
correctly_rounded_divide
dividednumerator / rate as a source written in PyTorch divides a state by a rate, or an array by a constant: IEEE division, whatever the backend’s own division does (XLA takes a division by a constant as a product with its rounded reciprocal, on a CPU and an A100 alike, row 1402).
effective_precisionThe precision a product multiplies at, resolved once: the one it states, else the run’s configured one (jax_default_matmul_precision, a run’s Numerics.matmul, generation.run_precision), which a route takes as the call’s declaration, honouring it or handing the call to a route that does, never replacing it with a backend’s own arithmetic.
exact_float32The precision a float32 product whose rounding decides what follows takes, a numerically critical one, a vocabulary head’s (the loss’s logits), a router’s or a recurrence’s state’s, each stating its reason where it calls this: the one the call states, else IEEE float32 (Precision.HIGHEST), stated over JAX’s configured precision, whose default runs TF32 on a GPU and one bf16 pass on a TPU, where the references multiply IEEE float32 (torch’s float32 matmul).
exact_productsWhether a route that multiplies dtype operands whole with float32 sums (an fp32 FMA, a 16-bit operand’s exact product, fp32 operands in bf16 parts that hold their 24 bits) computes the declaration effective_precision(precision): a precision level, each a floor (DEFAULT, HIGH, HIGHEST), or a dot algorithm whose products are whole for these operands (F32_F32_F32, and a 16-bit operand’s own BF16_BF16_F32 or F16_F16_F32); not an algorithm that narrows them (TF32’s, bf16 parts of an fp32 product) or widens them (F64), which a route hands to XLA, which computes it.
float32_dot_generaljax.lax.dot_general at precision, accumulated as Dew accumulates a CPU’s float32 products (accumulated): the product a flax layer forms (nn.Dense(dot_general=...)).
float32_einsumjnp.einsum at precision, JAX’s configured one where unstated, as float32_dot_general is a layer’s, accumulated alike, and bf16_einsum’s product at the BF16_BF16_F32 algorithm.
float32_matmullhs @ rhs at precision, JAX’s configured one where unstated, accumulated as float32_dot_general is, summed and returned in preferred_element_type where given.
float64_sumproduct(*operands), a sum of products, taken in float64 from its operands widened exactly and rounded once to dtype, under jax.enable_x64 for its own trace: the sum accumulated takes on a CPU under Dew’s wide arithmetic (WIDE_CPU_ARITHMETIC), with no platform of its own to choose.
fp32_result_dot_generalA flax layer’s dot_general for a head whose result stays at least fp32.
head_dot_generalA flax layer’s dot_general for a vocabulary head: a result at least fp32 whose operands are the compute dtype’s values, in both directions.
head_productjnp.einsum(subscripts, hidden, head) as a vocabulary head computes it, with accumulation and a result at least fp32.
iteration_dtypeThe dtype an optimizer’s matrix iteration (Muon’s and M3’s Newton-Schulz, dew.newton_schulz) multiplies in under the run’s numerics: in its fast forms, bfloat16 for any 16-bit compute dtype (the measured default under bf16-mixed: a Qwen3.5-4B decision step 844 ms against 1287 at TF32 and 4641 at IEEE float32 on an H100) and its compute dtype otherwise; at least float32 of it in its accurate ones; a float64 run iterates in float64.
l2normx over its L2 length along the last axis, x * rsqrt(sum(x**2) + eps) with eps added under the root, computed in at least fp32 (at_least_fp32, a float64 input’s in float64) and rounded once to x’s dtype: the normalization of FLA’s l2norm (the linear attentions’ queries and keys, modeling_qwen3_next.py:368-371) and of nGPT’s justnorm (its residual stream and weights).
numerics_ofA run’s numerics from what a flag, a record or a load names: a Numerics as it is, a preset’s name (NUMERICS_PRESETS, “default” for Numerics()), or the usual dtypes’ Lightning name (DTYPE_PRESETS) at the default precision and forms.
operand_precisionThe precision a product of dtype operands takes: the one stated, JAX’s configured one where none is, and for bf16 operands at the default precision the algorithm that keeps a bf16 dot bf16 in both directions.
paired_sumAn empty running sum of shape that is a double-float pair on every backend, the plain sum beside it (added_to, running_total): for partials of a product Dew splits where its reference takes the product whole, so the sum’s error is the partials’ own, whether XLA takes the addition into a GEMM’s epilogue or cuBLAS splits a product’s sum by its shape (row 1725).
precision_namesThe canonical names a jax.lax.PrecisionLike spells, unordered.
products_dtypeThe dtype operands multiply in, by jax’s implicit promotion; None where an 8-bit float takes part, which jax promotes with nothing and whose product has no precision to resolve.
rms_statisticsMean square and inverse RMS in at least fp32.
rms_weightedAn RMS norm’s arithmetic past its statistics: x times its rows’ inverse RMS (rms_statistics’s, or a kernel’s own), times weight, times activation(gate) (activation_and_slope: Qwen3-Next’s gated norm), in that order and in x’s dtype, which the caller widens (at least fp32) and rounds once after.
rounded_operandThe value x takes in dtype, held in x’s own dtype (_rounded), with a straight-through derivative (dew.nn.derivatives.surrogate): its tangent is x’s, never rounded a second time.
rounded_tox rounded to dtype’s precision and held in x’s own dtype: where a torch computation stores a tensor of dtype.
rounds_productsWhether the run’s declaration lets a native route round an fp32 product’s operands (TF32, or bf16 parts): its fast forms (Numerics.forms) at a configured precision of no more than the default (asks_default_precision); a run at ‘highest’ or a dot algorithm keeps them whole whatever its forms.
rounds_to_bf16Whether an fp32-result product under this compute dtype and precision multiplies bf16 operands (operand_precision).
running_sumAn empty running sum of shape (added_to, running_total): [3, *shape], a double-float pair and the plain sum beside it, where loops compensate (compensates_loops), else [1, *shape], the plain sum alone.
running_totalA running sum’s value (running_sum, or its leading axis’s parts): a pair’s rounded once (compensated_total), else the sum.
scaledx * factor with the factor kept in fp32 and only the product rounded to x’s dtype, which is what torch’s bf16_tensor * python_float does (fp32 opmath) and what Gemma’s embed_scale here does.
scaled_logitsA vocabulary head’s per-column terms after its product’s rounding, in fp32 or wider: each column times its scale (nGPT’s per-token logit gain), then plus its bias.
span_meansThe mean of values [B, L, D] over each [..., 2] (start, end) span of its row.
spreadvalue broadcast to shape, whose transpose, the sum of a cotangent over every position the value spread over (a bias’s gradient), a CPU running wide arithmetic (WIDE_CPU_ARITHMETIC, Dew’s tests) takes in float64 and rounds once, as it sums a float32 product (accumulated): there the broadcast is the outer product of ones and the value, whose transpose is a product.
sqrt_or_zerosqrt(square), and zero with a zero gradient where square is zero: sqrt’s infinite derivative there would turn a masked row’s zero upstream gradient into NaN for every parameter it touches.
stochastic_bf16fp32 x rounded to bf16 up in magnitude with the probability of its dropped low 16 bits: noise’s low 16 bits (uint32, uniform) added below the kept bits and the sum truncated, which carries into them exactly with that probability.
two_productDekker’s rounded product and its residual, with a Veltkamp significand split.
two_sumleft + right rounded, and exactly what the rounding dropped (Knuth’s TwoSum).
weighted_totalWeighted total and mass along axis (an int, a tuple, or None for every axis), in x’s dtype, each summed once rounded (compensated_sum), so a loss over many tokens rounds as its exact sum does.
wide_cpu_arithmeticWhether this process runs wide CPU arithmetic (WIDE_CPU_ARITHMETIC).
widened_reductionreduction(*operands), a reduction returning arrays: the plain one, but in a process running wide CPU arithmetic (WIDE_CPU_ARITHMETIC, Dew’s tests) a CPU takes it over the float32 operands widened exactly to float64, each float64 result rounded once back to float32, as accumulated takes a product’s sums: XLA’s CPU compiler runs a reduction with a combine of its own as one loop over the row, rounding at each step.

attribute source

ACCURATE = Numerics(DTYPE_PRESETS['32-true'], matmul='highest', forms='accurate')

Accurate, scientific work at its cost: float32 throughout, every product that states no precision in IEEE float32 and each op’s accurate form.

attribute source

DTYPE_PRESETS = {'bf16-mixed': Dtypes(), '32-true': Dtypes(compute='float32'), 'bf16-true': Dtypes(params='bfloat16', output='bfloat16'), '16-mixed': Dtypes(compute='float16')}

The usual dtypes by Lightning’s names: float32 parameters computing in bf16 by default, float32 throughout, bf16 throughout, and float16 compute over float32 parameters.

attribute source

FP32_SUMS = 'F32_F32_F32'

JAX’s dot algorithm of fp32 operands summed in fp32: as a process’s or a scope’s default matmul precision, the statement that its fp32 products sum as an accelerator sums them, which accumulated honors on a CPU.

attribute source

L2NORM_EPS = 1e-06

The epsilon l2norm adds under its square root: FLA’s, and Dory’s justnorm’s.

attribute source

MATMUL_PRECISIONS = ('default', 'high', 'highest')

JAX’s matmul precision levels (jax_default_matmul_precision): “default” TF32 for a float32 product on a GPU and one bf16 pass on a TPU, “highest” IEEE float32.

attribute source

NUMERICS_PRESETS = {'accurate': ACCURATE}

The named run numerics beside the default (Numerics()).

attribute source

ROUNDING_STREAM = 1919907182

The stream stochastic rounding draws from: what a training step folds its key with (dew.objectives.base.training_rngs), and what a served row’s rounding key starts from (dew.sampling.text.rounding_keys), apart from dropout’s and sampling’s.

attribute source

WIDE_CPU_ARITHMETIC = 'DEW_WIDE_CPU_ARITHMETIC'

The environment variable that, set to 1, has a process’s CPU programs take the arithmetic Dew’s tests hold a CPU to: float32 products summed in float64 and rounded once (accumulated, spread), a loop’s running sum of them compensated so it rounds once too (compensates_loops), and a float32 reduction taken in float64 and rounded once (widened_reduction). Read as each program is traced; set by the tests’ environment (tests/lane_environment.py), never by the library, which without it takes none of these forms on any platform. Each such site chooses the wide form by the platform the program is lowered for (platform_dependent, or for a loop’s carry the platform its code is chosen for), so a test that lowers for an accelerator keeps the target’s.

dataclass source

class Dtypes(
params: str = 'float32',
compute: str = 'bfloat16',
output: str = 'float32',
)

A run’s dtypes, three as Flax casts them: the parameters’ (its layers’ param_dtype, and the optimizer’s moments’), the compute’s (their dtype) and the outputs’ (the logits and losses a model returns). A source the run keeps frozen is stored as its checkpoint stores it, unless the parameters’ dtype is narrower (dew.config.RunConfig.source_keywords).

compute: str

bf16 by default, Lightning’s bf16-mixed, where Flax computes in its parameters’ float32: a run’s matmuls run on the tensor cores’ bf16 rate, its float32 masters and moments keep the update exact, and the float32 products a run needs say so (exact_float32); “32-true” computes in float32.

dataclass source

class Numerics(
dtypes: Dtypes = Dtypes(),
matmul: str | jax.lax.DotAlgorithmPreset = 'default',
forms: Literal['fast', 'accurate'] = 'fast',
)

A run’s numerics, set once and recorded with it, which its model, optimizer and trainer read: its dtypes (Dtypes, which a model takes as Flax’s param_dtype and dtype and its optimizer as its moments’; DTYPE_PRESETS names the usual ones by Lightning’s names), the precision every product that states none multiplies at (JAX’s own jax_default_matmul_precision level, or a jax.lax.DotAlgorithmPreset, which a record holds by its name), and each op’s form, its kernels’ fast or accurate arithmetic (dew.nn.kernels.generation.run_precision). A precision a call or a model states still wins, and determinism is a setting of its own. The default is bf16-mixed at JAX’s default precision in the fast forms; ACCURATE names the scientific one (NUMERICS_PRESETS).

function source

def accumulated(
product: Callable[..., jax.Array],
*operands=(),
preferred_element_type: DTypeLike | None = None,
) -> jax.Array

product(*operands, preferred_element_type=...), a sum of products of operands (a dot, an einsum, a matmul): the plain product, but in a process running wide CPU arithmetic (WIDE_CPU_ARITHMETIC, Dew’s tests), where they multiply in float32 (products_dtype) a CPU sums the products in float64 and rounds the sum once to the dtype the product returns. Each fp32 product is exact in float64, so the result is within half an fp32 ulp of the float64 sum at every shape, where XLA’s CPU dot chooses its fp32 accumulation by shape, instruction set and release (jax 0.12’s sums twice numpy’s error from float64 at widths where jax 0.11’s summed 0.6 of it). It costs a CPU product about twice the time (2048-cubed 56 to 126 ms on armada’s EPYC); a GPU or TPU product, and any whose products are 16-bit, are the product itself. float64 runs on these ops alone, under jax.enable_x64 for their own trace, which nothing outside reads, and their gradients and tangents are float64 sums too. An operand that is not an array (a quantized one, Qwix’s QArray) leaves the product to its own library, whose dequantized sum keeps XLA’s accumulation.

A library that swaps JAX’s products for its own while a model runs (Qwix, quantizing) owns them: the product is its, as flax’s own layers’ would be. One whose trace calls a backward of its own (a jax.custom_vjp, Qwix’s when it quantizes a model directly) is left to it whole, its forward and its backward, read from the trace that gives the product’s dtype (_own_backward).

The wide sum is Dew’s test setting, not a user option: a process without WIDE_CPU_ARITHMETIC traces the plain product alone. Under it, it is real CPU execution’s policy alone, chosen by the platform the program is lowered for (platform_dependent), which keys every compiled program: a step a CPU lowers for a GPU target (dew.training.trainer.Trainer.lower_for, lowering_platforms=('cuda',)) takes the target’s fp32 sum, and a Pallas kernel’s body, interpreted on a CPU to stand in for a GPU, never forms its products here. A computation that stands for such a target, a reference for a kernel interpreted on a CPU, states its fp32 sums with JAX’s dot algorithm for them, with jax.default_matmul_precision(FP32_SUMS), which jit’s cache keys as it keys every precision.

function source

def activation_and_slope(gate, activation: str) -> tuple[jax.Array, jax.Array]

An fp32 gate’s act(gate) and its derivative, for ‘silu’ or ‘sigmoid’.

silu’s derivative s + g s (1 - s), with s = sigmoid(g), cancels to zero near g = -1.28; formed as the product rule forms it, s + g (s (1 - s)), its worst element there is half (g s)(1 - s) + s’s, as XLA’s is.

function source

def added_to(running: Running, term: jax.Array) -> Running

running (running_sum) with term added: through compensated_add where it is a pair. A running sum held as its leading axis’s parts (a tuple) stays one, as a kernel whose layouts hold no unit axis carries it.

function source

def asks_default_precision(
precision: jax.lax.PrecisionLike,
*,
configured: bool = False,
) -> bool

Whether precision asks for no more than the default. With configured, an unset precision reads jax_default_matmul_precision, the default a product without one gets.

function source

def at_least_fp32(dtype: DTypeLike | None) -> jnp.dtype

The dtype arithmetic Dew keeps from rounding below float32 runs in: float32 for float32, every narrower float and a module’s unset dtype (None), and dtype itself where it is wider. Norm statistics, softmaxes, rotary and time-embedding angles, the vocabulary head’s product and loss reductions read it, so a bfloat16 model reduces in float32 and a float64 model computes in float64 throughout, not with its float32 twin’s roundings. An 8-bit float, which promotes implicitly to nothing, reads float32 as every narrower float does.

function source

def at_precision(
precision: jax.lax.PrecisionLike,
) -> Callable[[Callable[..., Output]], Callable[..., Output]]

fn traced, forward and backward, at precision, the default included: a product that takes no precision of its own (a Pallas dot, jax.nn’s xla attention) reads it from the configuration at trace time (jax.default_matmul_precision), and a custom derivative’s backward is traced after the forward returns, so it is held under the same setting by hand. An unset precision is the configured one, which forward and backward read alike: fn itself. A precision no config names (configurable_precision) is refused.

A fused attention route under a call’s stated precision (attention._attention_kernel), and Mosaic’s TPU kernels at the default: they refuse a 16-bit matmul at HIGHEST (“Bad lhs type”); 16-bit dots are exact at any precision, and an fp32 dot takes one bf16 pass on a TPU, so splash’s P·V rounds P to bf16 as every default-precision run does (attention_tpu).

function source

def bf16_dot_general(lhs: jax.Array, rhs: jax.Array, dimension_numbers) -> jax.Array

jax.lax.dot_general(lhs, rhs, dimension_numbers) multiplied as XLA’s BF16_BF16_F32 dot algorithm states the arithmetic: the operands rounded to bf16 (to nearest even), each product exact and summed in fp32 to an fp32 result. Backward, the cotangent rounds to bf16 once and each operand’s gradient is the same product of it and the other operand, held in the operand’s dtype; forward mode takes the product’s own tangent (linear_tangent). The one product Dew runs at that algorithm: the dot family’s (float32_dot_general, float32_einsum) and a bf16 head’s.

Its operands are bf16 arrays multiplied at the default precision, whatever a scope configures (an F16 preset would narrow a bf16 operand to fp16), not wider ones under the algorithm; a CPU, which runs no bf16 dot at some layouts, multiplies their values in fp32, exactly. On an A100, XLA’s dot under the algorithm with a vector side (one row or one column), or below its GEMM rewrite threshold (6 x 3 x 8), did not round fp32 operands to nearest even, and products of exact bf16 values came out off round-to-nearest-even at their ties; at a vector side bf16 operands under the algorithm missed too, at the default precision and at the highest. With bf16 operands and no algorithm every shape was exact (node receipts SF-a100-tie-probe3 and 4). A one-token tile of the chunked head and a one-row decode are such dots.

function source

def bf16_einsum(subscripts: str, *operands=(), **kwargs={}) -> jax.Array

jnp.einsum(subscripts, *operands) in fp32, its every product bf16_dot_general’s: einsum plans the contraction (its path, broadcasts, and each operand’s own sums, diagonals and transposes, which form no product and run in fp32) and hands each pairwise product to bf16_dot_general as its _dot_general. kwargs are einsum’s own (optimize, out_sharding).

function source

def bf16_logits(logits: jax.Array) -> jax.Array

fp32 logits rounded to bf16 and held in fp32, as rounded_to rounds them, written as bf16 under CUDA.

There the cast joins the product’s epilogue and each reader widens in its own fusion, behind a barrier xla_allow_excess_precision cannot see through. reduce_precision instead stayed a kernel of its own after the Triton head, writing and reading the fp32 logits in full when a serving step’s draw read them three times: 0.16 ms of a step at 128 rows of Qwen3-0.6B on an RTX 4080, the same bits (docs/performance.md).

function source

def bf16_product(subscripts: str, hidden: jax.Array, head: jax.Array) -> jax.Array

jnp.einsum(subscripts, hidden, head) multiplied as bf16 and its result rounded to bf16, held in fp32: the one product a bf16 head computes (head_product) and a tiled head computes over each tile, so the two round each logit alike.

function source

def biased(product: jax.Array, bias: jax.Array) -> jax.Array

product plus bias broadcast over its leading axes, in the dtype their addition promotes to, as flax’s layer adds them. Rounding the product and the bias once together, as torch’s addmm does, wrote an fp32 product and a pass of its own on every biased layer (a DiT-B step 17.5% slower on an A100), for a bf16 output 1.37 times torch’s rounding, which the parity rule holds already.

JAX transposes a broadcast into a sum in the broadcast operand’s dtype (jax-ml/jax#30310), so a bfloat16 bias added to bfloat16 products sums its gradient over every position in bfloat16, where torch’s addmm sums its in float32: Qwen 2’s projection biases, summed over 8192 tokens, came out up to 3.75 times further from float64 than torch’s own bfloat16 gradients. Broadcast at fp32, the gradient sums at fp32 and rounds once into the bias. A CPU running Dew’s wide arithmetic (WIDE_CPU_ARITHMETIC) sums it in float64, as it sums a float32 product (spread).

function source

def capped_logits(logits: jax.Array, softcap, temperature: float = 1.0) -> jax.Array

Softcapped logits over a sampling temperature, which an engine applies to what the model’s head returns.

function source

def compensated_add(left, right)

Add two double-float values, (hi, lo) pairs whose sum is the value: hi is the sum rounded and lo what the rounding dropped (the his’ TwoSum, the los added to its remainder, then the pair’s own TwoSum), so a running sum keeps fp32’s precision squared and its hi is the running value rounded once, where a pair left unrenormalised keeps the plain running sum in hi, every addition’s rounding in it. XLA reassociates no float arithmetic unless fast math is on; with it, lo cancels to zero and the sum is the plain one.

function source

def compensated_difference(later, earlier)

later - earlier of two double-float values, rounded once: the halves’ differences summed, so the difference of two near values (a range of a running sum) keeps its own precision, where the his’ difference alone carries their whole magnitude’s rounding: a one-token range of a decay’s running sum near -78 off by 4e-6. Broadcasts as the halves do.

function source

def compensated_sum(values: jax.Array, axes: tuple[int, ...]) -> jax.Array

The sum of values over axes, rounded once: one XLA reduction whose accumulator is a double-float (hi, lo) pair combined by compensated_add, so it partitions as any reduction does, and what the roundings dropped is added back at the end.

One plain reduction over a loss’s every axis rounds as it goes; on 6144 JEPA squares it left the sum one float32 step above the correctly rounded one, and the loss a float32 ulp from float64, where numpy’s float32 mean of the same squares was correctly rounded. A narrower float adds in float32 and returns in its own dtype, as jnp.sum does; an integer sum is jnp.sum’s. The derivative is jnp.sum’s.

function source

def compensated_total(pair: tuple[jax.Array, jax.Array], plain: jax.Array) -> jax.Array

A compensated sum’s value, its (hi, lo) pair rounded once, or the plain sum of the same values where that reaches an infinity: +-inf or NaN as jnp.sum’s is, where every residual past the infinity is inf - inf. compensated_add itself stays arithmetic alone, so a tangent’s scan through it stays linear; a running sum that can overflow keeps its plain sum beside the pair and reads its value here.

function source

def compensates_loops() -> bool

Whether a running sum of float32 products carried across a loop keeps what each addition drops (compensated_add), so the loop’s sum rounds once as one product’s does: where a CPU sums each product in float64 (accumulated), in a process running Dew’s wide CPU arithmetic (WIDE_CPU_ARITHMETIC). A carry’s shape is fixed when its loop traces, so this reads the platform a call’s code is chosen for (dew.nn.kernels.generation.backend), as a CPU’s attention routing does; an accelerator’s running sum is the plain one, as its products’ sums are float32’s.

function source

def configurable_precision(precision: jax.lax.PrecisionLike) -> bool

Whether jax.default_matmul_precision can name precision: one precision or dot algorithm preset, unset or default included; not a pair of precisions or a dot algorithm.

function source

def configured_precision(precision: jax.lax.PrecisionLike)

A context in which a product that takes no precision of its own (jax.nn’s xla attention) multiplies at precision, through jax.default_matmul_precision; an unset or default precision changes nothing, a floor the configured one meets. A precision no config names (configurable_precision) is refused.

attribute source

correctly_rounded_divide = contracted('test_the_precision_seams_hold_every_order')(declared_tangent(Protected(), _correctly_rounded_divide, _divide_jvp))

function source

def divided(numerator, rate)

numerator / rate as a source written in PyTorch divides a state by a rate, or an array by a constant: IEEE division, whatever the backend’s own division does (XLA takes a division by a constant as a product with its rounded reciprocal, on a CPU and an A100 alike, row 1402). A float32, bfloat16 or float16 quotient is correctly_rounded_divide’s of the operands widened to float32, rounded to its dtype, as PyTorch’s float32 opmath gives it, under any compiler flags. A float64 one is the plain division.

function source

def effective_precision(precision: jax.lax.PrecisionLike) -> jax.lax.PrecisionLike

The precision a product multiplies at, resolved once: the one it states, else the run’s configured one (jax_default_matmul_precision, a run’s Numerics.matmul, generation.run_precision), which a route takes as the call’s declaration, honouring it or handing the call to a route that does, never replacing it with a backend’s own arithmetic.

function source

def exact_float32(precision: jax.lax.PrecisionLike) -> jax.lax.PrecisionLike

The precision a float32 product whose rounding decides what follows takes, a numerically critical one, a vocabulary head’s (the loss’s logits), a router’s or a recurrence’s state’s, each stating its reason where it calls this: the one the call states, else IEEE float32 (Precision.HIGHEST), stated over JAX’s configured precision, whose default runs TF32 on a GPU and one bf16 pass on a TPU, where the references multiply IEEE float32 (torch’s float32 matmul). Every other product follows JAX’s configured precision and the run’s numerics (dew.nn.kernels.generation.run_precision).

function source

def exact_products(precision: jax.lax.PrecisionLike, dtype: DTypeLike) -> bool

Whether a route that multiplies dtype operands whole with float32 sums (an fp32 FMA, a 16-bit operand’s exact product, fp32 operands in bf16 parts that hold their 24 bits) computes the declaration effective_precision(precision): a precision level, each a floor (DEFAULT, HIGH, HIGHEST), or a dot algorithm whose products are whole for these operands (F32_F32_F32, and a 16-bit operand’s own BF16_BF16_F32 or F16_F16_F32); not an algorithm that narrows them (TF32’s, bf16 parts of an fp32 product) or widens them (F64), which a route hands to XLA, which computes it.

function source

def float32_dot_general(
lhs,
rhs,
dimension_numbers,
precision: jax.lax.PrecisionLike = None,
preferred_element_type: DTypeLike | None = None,
**kwargs={},
) -> jax.Array

jax.lax.dot_general at precision, accumulated as Dew accumulates a CPU’s float32 products (accumulated): the product a flax layer forms (nn.Dense(dot_general=...)). Unstated, JAX’s configured precision decides (jax_default_matmul_precision), as it does a flax layer’s. At the BF16_BF16_F32 algorithm it is bf16_dot_general’s product.

function source

def float32_einsum(
subscripts: str,
*operands=(),
precision: jax.lax.PrecisionLike = None,
preferred_element_type: DTypeLike | None = None,
**kwargs={},
) -> jax.Array

jnp.einsum at precision, JAX’s configured one where unstated, as float32_dot_general is a layer’s, accumulated alike, and bf16_einsum’s product at the BF16_BF16_F32 algorithm.

function source

def float32_matmul(
lhs,
rhs,
precision: jax.lax.PrecisionLike = None,
preferred_element_type: DTypeLike | None = None,
) -> jax.Array

lhs @ rhs at precision, JAX’s configured one where unstated, accumulated as float32_dot_general is, summed and returned in preferred_element_type where given.

function source

def float64_sum(
product: Callable[..., jax.Array],
*operands=(),
dtype: DTypeLike,
) -> jax.Array

product(*operands), a sum of products, taken in float64 from its operands widened exactly and rounded once to dtype, under jax.enable_x64 for its own trace: the sum accumulated takes on a CPU under Dew’s wide arithmetic (WIDE_CPU_ARITHMETIC), with no platform of its own to choose. Its products run at HIGHEST precision, whatever fp32 algorithm the caller bound, which would narrow float64 operands back to float32.

function source

def fp32_result_dot_general(precision: PrecisionLike = None)

A flax layer’s dot_general for a head whose result stays at least fp32.

The layer has already promoted both operands when this runs - to its own dtype, or to the activations’ where it carries none - so the compute dtype is the operands’ own. Only the accumulation and the result widen, to at_least_fp32 of it.

function source

def head_dot_general(dtype: Dtype | None, precision: PrecisionLike = None)

A flax layer’s dot_general for a vocabulary head: a result at least fp32 whose operands are the compute dtype’s values, in both directions.

dtype is the model’s compute dtype, not the layer’s: the layer promotes the states to at least fp32. Under bf16 compute at the default precision both operands multiply as bf16 and the logits are bf16 values (head_product); otherwise the product is the layer’s own, in the operands’ dtype, at the precision stated, IEEE float32 where none is (exact_float32): its logits feed the loss’s logsumexp and the sampler, logits tens wide that TF32’s part in a thousand moves.

function source

def head_product(
subscripts: str,
hidden: jax.Array,
head: jax.Array,
precision: PrecisionLike = None,
*,
compute: DTypeLike | None = None,
) -> jax.Array

jnp.einsum(subscripts, hidden, head) as a vocabulary head computes it, with accumulation and a result at least fp32.

The product follows the compute dtype, as torch autocast and MaxText (logits_dot_in_fp32=False) run it: bf16 states at the default precision multiply the head as bf16 and the logits round to bf16, held in fp32, with the cotangent rounded once alike. Anything else multiplies fp32 states by the head as stored. The rounding cost nothing measurable in 2000-step wikitext-103 runs (validation loss within 4.5e-4 and 1.6e-3 of fp32 logits, below seed noise) and saves 13% of a 3-layer decoder’s step. It does move layout parity (a 4 x RTX 3090 bf16 MoE read 5758 of its bound against 0.41 in fp32), so a run comparing layouts or resuming onto another mesh sets matmul_precision “highest”, whose head is fp32. compute, the model’s compute dtype where the states arrive in another, decides in their place.

function source

def iteration_dtype() -> jnp.dtype

The dtype an optimizer’s matrix iteration (Muon’s and M3’s Newton-Schulz, dew.newton_schulz) multiplies in under the run’s numerics: in its fast forms, bfloat16 for any 16-bit compute dtype (the measured default under bf16-mixed: a Qwen3.5-4B decision step 844 ms against 1287 at TF32 and 4641 at IEEE float32 on an H100) and its compute dtype otherwise; at least float32 of it in its accurate ones; a float64 run iterates in float64. Never float16, whose range holds neither the iteration’s Frobenius norm past 256 nor optax’s 1e-8 under it, which rounds to zero there, so a zero update (a zero-initialized output projection’s upstream matrices at the first step) divides zero by zero.

function source

def l2norm(x: jax.Array, eps: float = L2NORM_EPS) -> jax.Array

x over its L2 length along the last axis, x * rsqrt(sum(x**2) + eps) with eps added under the root, computed in at least fp32 (at_least_fp32, a float64 input’s in float64) and rounded once to x’s dtype: the normalization of FLA’s l2norm (the linear attentions’ queries and keys, modeling_qwen3_next.py:368-371) and of nGPT’s justnorm (its residual stream and weights).

function source

def numerics_of(given: Numerics | str) -> Numerics

A run’s numerics from what a flag, a record or a load names: a Numerics as it is, a preset’s name (NUMERICS_PRESETS, “default” for Numerics()), or the usual dtypes’ Lightning name (DTYPE_PRESETS) at the default precision and forms.

function source

def operand_precision(
dtype: Dtype | None,
precision: jax.lax.PrecisionLike = None,
) -> jax.lax.PrecisionLike

The precision a product of dtype operands takes: the one stated, JAX’s configured one where none is, and for bf16 operands at the default precision the algorithm that keeps a bf16 dot bf16 in both directions. A caller that asked for more than the default keeps what it asked for, and so does a run whose configured level asks for more where the call states none (a run’s Numerics.matmul, generation.run_precision), and a GPU older than sm80, which rejects the algorithm at run time (dew.nn.kernels.generation.BF16_GPU). A configured dot algorithm preset is no level a run picks but how dots that state none sum (check 1’s arms state F32_F32_F32), which the algorithm’s fp32 sums already are, so it keeps the algorithm.

function source

def paired_sum(shape: tuple[int, ...], dtype: DTypeLike) -> jax.Array

An empty running sum of shape that is a double-float pair on every backend, the plain sum beside it (added_to, running_total): for partials of a product Dew splits where its reference takes the product whole, so the sum’s error is the partials’ own, whether XLA takes the addition into a GEMM’s epilogue or cuBLAS splits a product’s sum by its shape (row 1725).

function source

def precision_names(precision: jax.lax.PrecisionLike) -> frozenset[str]

The canonical names a jax.lax.PrecisionLike spells, unordered.

JAX’s alias is None, a string, a jax.lax.Precision, a dot algorithm preset or a dot algorithm, or a pair of precisions; flax’s is the same without the algorithms. A string is read as JAX reads jax_default_matmul_precision: a dot algorithm preset’s name is that preset (‘F32_F32_F32’), any other goes through jax.lax.Precision, so ‘tensorfloat32’ is ‘HIGH’ and ‘bfloat16’ is ‘DEFAULT’; a preset keeps its own name, and a dot algorithm, which has none, is named by its fields.

function source

def products_dtype(*operands=()) -> jnp.dtype | None

The dtype operands multiply in, by jax’s implicit promotion; None where an 8-bit float takes part, which jax promotes with nothing and whose product has no precision to resolve.

function source

def rms_statistics(
x: jax.Array,
epsilon: float,
*,
axis=-1,
keepdims: bool = True,
count: int | None = None,
) -> tuple[jax.Array, jax.Array]

Mean square and inverse RMS in at least fp32. A padded kernel tile supplies its logical count after zeroing padding; an ordinary array uses its reduction dimensions. Both means multiply by the reciprocal count, as XLA’s mean does. Callers own when the inverse, normalized values and learned scale round. The count’s reciprocal is a Python float, where jnp.reciprocal of the count’s array is an integer power, which a Pallas Triton kernel cannot lower on a folded constant.

function source

def rms_weighted(
x: jax.Array,
inverse: jax.Array,
*,
weight: jax.Array | None = None,
gate: jax.Array | None = None,
activation: str = 'silu',
rounded: Callable[[jax.Array], jax.Array] | None = None,
) -> jax.Array

An RMS norm’s arithmetic past its statistics: x times its rows’ inverse RMS (rms_statistics’s, or a kernel’s own), times weight, times activation(gate) (activation_and_slope: Qwen3-Next’s gated norm), in that order and in x’s dtype, which the caller widens (at least fp32) and rounds once after. rounded rounds the normalized rows, before any weight multiplies them (Llama’s and Qwen3’s cast first, RMSNorm.scale_after_cast; a norm computing in its input’s dtype). The fused norm kernel, the recurrence’s gated output norm, the portable norm and the portable gated norm compute it alike, each keeping its own statistics, memory schedule and derivative.

attribute source

rounded_operand = contracted('test_the_precision_seams_hold_every_order')(surrogate(_rounded, lambda dtype, primals, tangents, value: tangents[0], nondiff_argnums=(1,)))

The value x takes in dtype, held in x’s own dtype (_rounded), with a straight-through derivative (dew.nn.derivatives.surrogate): its tangent is x’s, never rounded a second time.

function source

def rounded_to(x: jax.Array, dtype: Dtype) -> jax.Array

x rounded to dtype’s precision and held in x’s own dtype: where a torch computation stores a tensor of dtype. Under jit XLA’s GPU default xla_allow_excess_precision deletes a narrowing cast a widening one follows; reduce_precision keeps the rounding but flushes float16 subnormals, so float16 rounds by the cast behind an optimization barrier (rounded_operand). The derivative rounds where the value does, as a torch tensor of dtype’s gradient does.

function source

def rounds_products() -> bool

Whether the run’s declaration lets a native route round an fp32 product’s operands (TF32, or bf16 parts): its fast forms (Numerics.forms) at a configured precision of no more than the default (asks_default_precision); a run at ‘highest’ or a dot algorithm keeps them whole whatever its forms.

function source

def rounds_to_bf16(dtype: Dtype | None, precision: PrecisionLike = None) -> bool

Whether an fp32-result product under this compute dtype and precision multiplies bf16 operands (operand_precision).

function source

def running_sum(shape: tuple[int, ...], dtype: DTypeLike) -> jax.Array

An empty running sum of shape (added_to, running_total): [3, *shape], a double-float pair and the plain sum beside it, where loops compensate (compensates_loops), else [1, *shape], the plain sum alone. Its leading axis is the form’s, so a loop slices and updates the sum’s own axes one past theirs.

function source

def running_total(running: Running) -> jax.Array

A running sum’s value (running_sum, or its leading axis’s parts): a pair’s rounded once (compensated_total), else the sum.

function source

def scaled(x: jax.Array, factor: float) -> jax.Array

x * factor with the factor kept in fp32 and only the product rounded to x’s dtype, which is what torch’s bf16_tensor * python_float does (fp32 opmath) and what Gemma’s embed_scale here does. Rounding 0.22 to bf16 first would shrink every product by a systematic 0.12%.

function source

def scaled_logits(
logits: jax.Array,
scale: jax.Array | None,
bias: jax.Array | None,
) -> jax.Array

A vocabulary head’s per-column terms after its product’s rounding, in fp32 or wider: each column times its scale (nGPT’s per-token logit gain), then plus its bias.

function source

def span_means(values: jax.Array, spans: jax.Array) -> jax.Array

The mean of values [B, L, D] over each [..., 2] (start, end) span of its row.

spans is [B, ...]; an empty span, a padding slot’s, gives zeros. A token no span of its row reads is selected away before the product, so padding’s NaN or infinity never reaches a mean (the rule of weighted_total), and the sum is the product’s accumulated one.

function source

def spread(value: jax.Array, shape: tuple[int, ...]) -> jax.Array

value broadcast to shape, whose transpose, the sum of a cotangent over every position the value spread over (a bias’s gradient), a CPU running wide arithmetic (WIDE_CPU_ARITHMETIC, Dew’s tests) takes in float64 and rounds once, as it sums a float32 product (accumulated): there the broadcast is the outer product of ones and the value, whose transpose is a product. Everywhere else, and where accumulated leaves a product its own sum (FP32_SUMS, a quantized model’s trace), it is the broadcast, whose transpose JAX sums in the value’s dtype, as a user’s process sums it on every platform (ledger 3197). A value an Explicit axis places along an axis it spreads over (never a bias, whose leading axes biased adds) is broadcast: the outer product does not keep that placement on a dropped size-one axis.

function source

def sqrt_or_zero(square)

sqrt(square), and zero with a zero gradient where square is zero: sqrt’s infinite derivative there would turn a masked row’s zero upstream gradient into NaN for every parameter it touches.

function source

def stochastic_bf16(x, noise)

fp32 x rounded to bf16 up in magnitude with the probability of its dropped low 16 bits: noise’s low 16 bits (uint32, uniform) added below the kept bits and the sum truncated, which carries into them exactly with that probability. It does not saturate: a value past bf16’s largest finite rounds to infinity with its dropped bits’ probability, as rounding to nearest would. NaN stays NaN. The noise is its caller’s policy: a hash of the state’s own bits and the token (recurrence.round_to_bf16), of the step’s seed and the element’s index (dew._moments.stochastic_round_bf16).

function source

def two_product(
left,
right,
*,
bit_split: bool = False,
subtract: bool = False,
rounding: Callable[[jax.Array], jax.Array] | None = None,
) -> tuple[jax.Array, jax.Array]

Dekker’s rounded product and its residual, with a Veltkamp significand split.

Guidance keeps the multiply split, which also supports float64. bit_split truncates fp32 significands instead, avoiding splitter overflow in norm interpretation and quantizer division; near underflow the splits differ. subtract keeps division’s subtractive residual order and its signed zeros. Norm interpretation passes optimization_barrier as rounding to keep each product and residual step separate; the other callers allow fusion. With each arithmetic step rounded, the residual is exact where intermediates neither overflow nor underflow.

function source

def two_sum(left, right)

left + right rounded, and exactly what the rounding dropped (Knuth’s TwoSum).

function source

def weighted_total(
x: jax.Array,
weights: jax.Array,
axis=None,
) -> tuple[jax.Array, jax.Array]

Weighted total and mass along axis (an int, a tuple, or None for every axis), in x’s dtype, each summed once rounded (compensated_sum), so a loss over many tokens rounds as its exact sum does. Broadcastable weights mark excluded values with zero; those values are selected away before multiplication, so their NaNs and infinities cannot reach the sum. The caller owns its denominator policy.

function source

def wide_cpu_arithmetic() -> bool

Whether this process runs wide CPU arithmetic (WIDE_CPU_ARITHMETIC).

function source

def widened_reduction(
reduction: Callable[..., Sequence[jax.Array]],
*operands: jax.Array = (),
) -> tuple[jax.Array, ...]

reduction(*operands), a reduction returning arrays: the plain one, but in a process running wide CPU arithmetic (WIDE_CPU_ARITHMETIC, Dew’s tests) a CPU takes it over the float32 operands widened exactly to float64, each float64 result rounded once back to float32, as accumulated takes a product’s sums: XLA’s CPU compiler runs a reduction with a combine of its own as one loop over the row, rounding at each step. A process without the setting, or operands not float32, take the plain reduction on every platform, chosen by the platform a program is lowered for (platform_dependent).