Skip to content

dew.quantization

Quantization of a model’s matmuls through Qwix, for training, serving and loading.

Qwix (google/qwix, Apache 2.0) describes quantization as rules over module paths and applies them without editing the model. One call wraps the module, and the matmuls that run inside the wrapped methods run quantized.

In Dew that call is Quantization.apply. A caller builds its model as usual, then wraps it before the objective ever sees it. A direct qwix.quantize_model call works too: Dew’s own GEMM kernels ask Qwix’s public rule query (qwix.pallas.get_current_rule) and leave a matched product to Qwix. Quantize a model once, through either path. A direct call’s traces are not keyed apart from the plain model’s as Quantization.apply’s are (dew._cache.QUANTIZED_TRACE), so a jitted helper the two share needs Qwix’s disable_jit=True. A run that sets --trainer.quantization passes the objective to RunConfig.train instead, and the model the objective holds is wrapped there, before anything initialises it (dew.training.quantization.quantize_trunk). This module is the model’s: loading, adapters, training and serving read it, and it reads nothing of training.

Training is fake-quantized. The parameter tree keeps fp32 master weights with the same structure, so the checkpoint layout, the sharding derivation, the Muon parameter split and Hugging Face loading are unchanged. Only the forward and backward matmuls are quantized, with a straight-through estimator on the backward pass.

Serving stores the weights quantized. quantize_for_serving runs Qwix’s post-training quantization: the returned variables hold each matched kernel as int8 or fp8 values with their scales, and the returned module’s matmuls read them. TextGeneration.quantized applies it to a language model and TextToImage.quantized to an image task’s denoiser.

The vocabulary head stays fp32, like the rest of Dew’s fp32 zones. Its einsum runs in the objective’s chunked cross entropy, outside any model method Qwix wraps.

The Quantization fields follow MaxText’s quantization settings where Qwix has an equivalent. dtype is MaxText’s quantization for the dynamic-range forms, and patterns is its quant_cfg_path, written inline as the regexes Qwix matches. The backward fields are Qwix’s finer-grained version of the same idea.

Two MaxText settings have no equivalent, and dtype accepts neither. Static activation scaling (fp8_full) needs a calibration pass that Dew has no hook for, and nanoo_fp8 uses AMD-only kernels. KV-cache quantization is not a weight quantization: the cache’s own layout stores int8 or float8 keys and values (dew.nn.kv_cache).

Qwix comes with the quantization extra (pip install "dewml[quantization]"). It is imported inside the calls that need it, and without the package those calls raise an error that names the extra.

NameSummary
InputQuantizationThe quantization at a model’s Linear sites: a checkpoint’s input quantization, which a model loaded from a ModelOpt or compressed-tensors FP8 or NVFP4 checkpoint computes under (dew.interop.Pretrained), and a LoRA run’s frozen FP8 base (dew.lora.LoRA.fp8_base); a run records it.
QuantizationSays how a run quantizes its trunk matmuls, for Qwix’s provider.
QuantizedProductsHow the Qwix quantization a call runs under changes the products of a submodule and of the modules under it (quantized_products).
QuantizedTraceWhat a quantized model’s methods trace under (QUANTIZED_TRACE, which jit’s cache keys on): the caller’s opaque cache key, and the Quantization its provider follows (policy), or None for a provider whose rules Dew does not know.
decoded_storedvariables with each weight serving quantization stores quantized (Annotated, Qwix’s post-training annotation) decoded by Qwix’s public dequantize into the float its values and scales hold: the stored weight a layer that moves its weights starts from (dew.nn.memory’s W_0), every other leaf as it is.
moving_weightsTrace a module’s products at weights a layer moves from their stored values each call (an in-context memory’s network at the state its steps move from W_0): serving quantization multiplies them as quantized training does, each rounded to its rule’s dtype at each product and its gradient rounded as the rule’s backward fields ask (_serving_provider), as no stored quantized copy of a moving weight exists.
quantize_withQwix’s quantize_model(model, provider, methods=methods), each of methods tracing what it calls under key in QUANTIZED_TRACE (by default the provider’s kind and the provider): the way Dew runs a model Qwix quantizes with a provider of its own, so Qwix owns the products the thread tracing it forms (dew.nn.precision.accumulated) and no other thread’s, and its traces stay apart from the plain model’s.
split_columnsleaf split into one leaf a member along axis by widths, a packed projection’s kernel taken back into its members’ (dew.inference.projections.unpacked): a plain array, host or device, as np.split or jnp.split would; a Qwix QArray with its values split and each scale and zero point split as its own axis is, shared (one entry) kept whole by every member, per channel split at the same columns, per tile at the tiles’ (a member that would cut a tile is refused); and the serving tree’s wrapper of one (Qwix’s annotated array, array its field) as the QArray inside it, its recipe kept, where no other field of it holds an array that would need a split of its own.

dataclass source

class InputQuantization(
e4m3_scale: bool = True,
format: InputFormat = 'compressed-tensors',
sites: tuple[tuple[str, InputKind], ...] = (),
frozen_fp8: tuple[str, ...] = (),
measured_on: str | None = None,
)

The quantization at a model’s Linear sites: a checkpoint’s input quantization, which a model loaded from a ModelOpt or compressed-tensors FP8 or NVFP4 checkpoint computes under (dew.interop.Pretrained), and a LoRA run’s frozen FP8 base (dew.lora.LoRA.fp8_base); a run records it.

Each quantized Linear holds its scale, or its frozen FP8 kernel (FROZEN_FP8), in its own QUANTIZATION variables, so a scanned, staged or packed view of the layers keeps it, and a site without one computes as it is. This record holds what the files declare (NVFP4’s local-scale rounding and arithmetic, and which sites take their input in which format) and which frozen sites multiply in FP8, as measured on measured_on. It is not a Quantization: nothing trains quantized, and the weights already hold the values the source reader decoded.

e4m3_scale: bool

Whether NVFP4 sites round their local scales to E4M3.

format: InputFormat

Whose NVFP4 local-scale arithmetic the sites follow.

sites: tuple[tuple[str, InputKind], ...]

Each quantized Linear’s scope in the stored variables tree, /-joined, and its format: what init declares, so a model rebuilt from this record initializes the scales a checkpoint restores. The values live in the variables alone.

frozen_fp8: tuple[str, ...]

The frozen Dense sites whose products run in FP8 (frozen_fp8_dense), by their scope in the stored variables tree, /-joined: those whose FP8 forward and input gradient measured faster than bf16’s on measured_on (measured_fp8_sites). LoRA packs their kernels from the base it freezes, the one place a kernel is known never to train.

measured_on: str | None

The GPU generation frozen_fp8 was measured on (sm90): where its sites measured faster in FP8, which another generation may not repeat. The product is ordinary arithmetic on the stored codes and scales and runs on any generation; measure that GPU’s sites for its own speed (LoRA.fp8_base).

def layer_sites(layer: str) -> tuple[tuple[str, InputKind], ...]

The quantized sites under the stored scope layer (layers_3), relative to it, with their formats: what two otherwise like layers must share to run as one scanned body or one stage position.

def declared() -> Variables

The QUANTIZATION collection sites declare, every scale 1 until a checkpoint’s values replace it.

def quantizer(module: nn.Module) -> InputQuantizer | None

The bound module’s input quantizer, from the scale its variables hold, or None for a module that holds none.

def joined(model: nn.Module) -> nn.Module

model computing under this record’s frozen FP8 sites, and under the input quantization it already computes under, if any: one wrapper, so one interceptor, serves both.

def packed(frozen: Variables) -> Variables

The QUANTIZATION leaves of frozen_fp8, packed from the kernels the frozen tree holds at their scopes (frozen_fp8_leaves).

def apply(model: nn.Module) -> nn.Module

Return model quantizing each Linear’s input to its declared format and back with its scales, before the layer’s dot, and computing each Linear that holds a frozen FP8 kernel from it (frozen_fp8_dense).

Both run at the call of every module that holds a scale or a frozen kernel, under an interceptor apply installs, so every entry method of the model runs them, inside an adapter’s branch. An input’s dot is otherwise unchanged, so its dtype, accumulation precision, output sharding and the bias placement stay as the model defines them.

dataclass source

class Quantization(
dtype: QuantizedDtype = 'int8',
weight_only: bool = False,
patterns: tuple[str, ...] = ('.*',),
calibration: str = 'absmax',
tile_size: int | None = None,
bwd_qtype: QuantizedDtype | None = None,
bwd_stochastic_rounding: Rounding | None = None,
)

Says how a run quantizes its trunk matmuls, for Qwix’s provider.

dtype: QuantizedDtype

The dtype weights and activations quantize to, in the forward pass.

weight_only: bool

Whether to quantize only the weights and keep activations in the compute dtype.

Convolutions stay unquantized when this is set, because Qwix’s serving provider refuses a convolution whose activations stay in float.

patterns: tuple[str, ...]

Regexes over module paths that select what is quantized, in Qwix’s precedence order.

The first rule whose regex fully matches a module’s /-joined scope path wins. '.*mlp.*' quantizes the feed-forward blocks and leaves attention in fp32; the default quantizes every matmul of the wrapped methods.

calibration: str

The weight calibration method, in the form Qwix parses (absmax,0.8, for example). Dew keeps no copy of its grammar: Qwix refuses a method it does not know where it first quantizes a matched weight, not when this spec is built.

tile_size: int | None

The number of elements per tile for sub-channel scales along the contraction axis.

Unset keeps per-channel scales, which are coarser and cheaper.

bwd_qtype: QuantizedDtype | None

The dtype gradients quantize to in the backward pass; unset keeps them in the compute dtype.

bwd_stochastic_rounding: Rounding | None

The stochastic rounding mode for the quantized gradients; unset rounds deterministically.

Qwix draws from the stochastic_rounding RNG stream, which Dew’s objectives fold off each step’s key (dew.objectives.base.training_rngs); a custom objective passes it the same way.

def rule_for(scope: Sequence[str]) -> int | None

The index of the pattern that quantizes the module at scope (its path of names), the first whose regex fully matches the /-joined path as Qwix matches it, or None for a module no pattern selects.

def apply(model: nn.Module) -> nn.Module

Return a copy of model whose trunk matmuls train in this spec’s dtype.

The copy is an instance of Qwix’s subclass of the model’s class, with each of its entry_methods wrapped. Everything the registry, the objective and the checkpoint code read from the model therefore still works. Invalid settings were already refused when the spec was constructed, all but the calibration method, which Qwix parses where it first quantizes a matched weight. Without Qwix installed, the call raises an error that names the extra.

dataclass source

class QuantizedProducts(inputs: bool, weights: bool, stochastic: bool = False)

How the Qwix quantization a call runs under changes the products of a submodule and of the modules under it (quantized_products). inputs: their activations quantize, or their gradients do, so the submodule is no linear map of its input, which a layer that applies it to another operand (MLA’s absorbed kv_b_proj) needs it to be; a weight-only rule with float gradients leaves it linear, by its quantized matrix. weights: their weights quantize, so the submodule multiplies by no weight it holds, which a layer that moves its weights (an in-context memory’s recurrence) needs it to. stochastic: their gradients quantize and round stochastically, so a call that runs their backward draws the stochastic_rounding stream (an in-context memory’s inner steps).

dataclass source

class QuantizedTrace(key: Hashable, policy: Quantization | None)

What a quantized model’s methods trace under (QUANTIZED_TRACE, which jit’s cache keys on): the caller’s opaque cache key, and the Quantization its provider follows (policy), or None for a provider whose rules Dew does not know.

function source

def decoded_stored(variables: Variables) -> Variables

variables with each weight serving quantization stores quantized (Annotated, Qwix’s post-training annotation) decoded by Qwix’s public dequantize into the float its values and scales hold: the stored weight a layer that moves its weights starts from (dew.nn.memory’s W_0), every other leaf as it is.

function source

def moving_weights() -> Iterator[None]

Trace a module’s products at weights a layer moves from their stored values each call (an in-context memory’s network at the state its steps move from W_0): serving quantization multiplies them as quantized training does, each rounded to its rule’s dtype at each product and its gradient rounded as the rule’s backward fields ask (_serving_provider), as no stored quantized copy of a moving weight exists. The products of every other weight are as the provider forms them.

function source

def quantize_with(
model: nn.Module,
provider: QuantizationProvider,
*,
methods: Sequence[str] = ('__call__',),
key: Hashable | None = None,
policy: Quantization | None = None,
) -> nn.Module

Qwix’s quantize_model(model, provider, methods=methods), each of methods tracing what it calls under key in QUANTIZED_TRACE (by default the provider’s kind and the provider): the way Dew runs a model Qwix quantizes with a provider of its own, so Qwix owns the products the thread tracing it forms (dew.nn.precision.accumulated) and no other thread’s, and its traces stay apart from the plain model’s. key is the caller’s, opaque; policy is the Quantization the provider follows, whose rules a layer reads (quantized_products), or None for a provider whose rules Dew does not know.

function source

def split_columns(leaf, widths: Sequence[int], *, axis: int = -1) -> list

leaf split into one leaf a member along axis by widths, a packed projection’s kernel taken back into its members’ (dew.inference.projections.unpacked): a plain array, host or device, as np.split or jnp.split would; a Qwix QArray with its values split and each scale and zero point split as its own axis is, shared (one entry) kept whole by every member, per channel split at the same columns, per tile at the tiles’ (a member that would cut a tile is refused); and the serving tree’s wrapper of one (Qwix’s annotated array, array its field) as the QArray inside it, its recipe kept, where no other field of it holds an array that would need a split of its own. A leaf this process can materialize whole (fully addressable, or replicated with an addressable copy) splits on the host and returns host arrays; any other splits where it lives. The widths are positive and sum to leaf’s size along axis, or the split is refused.