dew.newton_schulz
Muon’s Newton-Schulz orthogonalization (dew.optim’s Muon and M3 iterate through it): optax’s own
iteration (orthogonalized_leaf) over each update’s matrices (as_matrices), a head’s alone where a run
names its width (headed), the matrices of a shape stacked and spread whole across the mesh’s devices
(orthogonalized), in the run’s numerics or a stated dot algorithm (ns_operands), at Jordan’s
coefficients or Polar Express’s (polar_express).
| Name | Summary |
|---|---|
NEWTON_SCHULZ | Muon’s quintic Newton-Schulz coefficients, Jordan’s (modded-nanogpt), optax’s default. |
POLAR_EXPRESS_TABLE | Polar Express’s quintic per iteration (arXiv 2505.16932, Appendix A’s coeffs_list), designed for singular values in [1e-3, 1], before its safety factor. |
Preconditioning | How optax’s Newton-Schulz scales a matrix before it iterates (optax.contrib.muon’s preconditioning). |
ns_operands | The dtype Muon’s Newton-Schulz takes its operands in: an algorithm’s own, where one states the iteration’s products (BF16_BF16_F32’s bfloat16, TF32_TF32_F32’s and F32_F32_F32’s float32, which a platform that has not the algorithm refuses in JAX’s own words, and a complex update, whose products no real algorithm holds, refuses by name); else the run’s numerics’ (dew.nn.kernels.generation.numerics), each product at its precision (Numerics.matmul), as dew.nn.precision.iteration_dtype declares it: in its fast forms bfloat16 for a 16-bit compute dtype (16-mixed’s float16 among them), as Jordan’s Muon (modded-nanogpt) and Moonlight (arXiv 2502.16982) iterate, and the compute dtype otherwise; at least float32 of it in its accurate ones, TF32 products on a GPU at the default precision and IEEE float32 under the accurate preset (precision="accurate"); complex64 for a complex update; and at least float32 of the iteration’s dtype for a spectral preconditioning, whose norm is an SVD of at least float32 operands (a float64 run’s in float64). |
orthogonalized | Each update numbers makes a stack of matrices of (as_matrices) orthogonalized by optax’s Newton-Schulz (orthogonalized_leaf) in the run’s numerics or algorithm (ns_operands), as Moonlight distributes it (arXiv 2502.16982, 3): the matrices of one shape gathered into one stack, spread across the mesh’s devices whole, each device iterating its own share, and handed back as the updates lie. |
orthogonalized_leaf | update orthogonalized by Muon’s steps quintic Newton-Schulz iterations at coeffs, one triple or a schedule of them, after preconditioning, optax’s own: its Muon transform with no momentum (scale_by_muon, beta 0 and no Nesterov step) hands back the update it is given orthogonalized, in its own dtype, bit for bit its orthogonalize_via_newton_schulz, which optax 0.2.8 does not export. |
polar_express | Polar Express’s coefficients for steps iterations as its code applies them (arXiv 2505.16932, Appendix A): each but the last triple’s polynomial taken at x / safety, (a / s, b / s^3, c / s^5), the first’s again for its input’s own scaling, X / (safety * ||X||_F), which optax’s iteration, over X / ||X||_F, then reproduces; the table’s first steps, its last repeated past it. |
NEWTON_SCHULZ
Section titled “NEWTON_SCHULZ”NEWTON_SCHULZ = (3.4445, -4.775, 2.0315)Muon’s quintic Newton-Schulz coefficients, Jordan’s (modded-nanogpt), optax’s default.
POLAR_EXPRESS_TABLE
Section titled “POLAR_EXPRESS_TABLE”POLAR_EXPRESS_TABLEPolar Express’s quintic per iteration (arXiv 2505.16932, Appendix A’s coeffs_list), designed for
singular values in [1e-3, 1], before its safety factor.
Preconditioning
Section titled “Preconditioning”Preconditioning = Literal['frobenius', 'spectral', 'aol', 'schatten']How optax’s Newton-Schulz scales a matrix before it iterates (optax.contrib.muon’s preconditioning).
ns_operands
Section titled “ns_operands”def ns_operands( dtype, preconditioning: Preconditioning = 'frobenius', algorithm: jax.lax.DotAlgorithmPreset | None = None,) -> jnp.dtypeThe dtype Muon’s Newton-Schulz takes its operands in: an algorithm’s own, where one states the
iteration’s products (BF16_BF16_F32’s bfloat16, TF32_TF32_F32’s and F32_F32_F32’s float32, which a
platform that has not the algorithm refuses in JAX’s own words, and a complex update, whose products no
real algorithm holds, refuses by name); else the run’s numerics’ (dew.nn.kernels.generation.numerics),
each product at its precision (Numerics.matmul), as dew.nn.precision.iteration_dtype declares it:
in its fast forms bfloat16 for a 16-bit compute dtype (16-mixed’s float16 among them), as Jordan’s Muon
(modded-nanogpt) and Moonlight (arXiv 2502.16982) iterate, and the compute dtype otherwise; at least
float32 of it in its accurate ones, TF32 products on a GPU
at the default precision and IEEE float32 under the accurate preset (precision="accurate");
complex64 for a complex update; and at least float32 of the iteration’s dtype for a spectral
preconditioning, whose norm is an SVD of at least float32 operands (a float64 run’s in float64).
The fast forms’ bfloat16, where optax iterates in its momentum’s own float32: the quintic brings the
singular values only near one, roughly within 0.7 to 1.2 (optax orthogonalize_via_newton_schulz), far
coarser than bfloat16’s rounding. A Qwen3.5-4B decision fine-tune on one H100 (node job LR-h100-muon-2752c
at 699d038ca, 8192 tokens a step) stepped in 844 ms with its iteration in bfloat16 (BF16_BF16_F32), 1287
ms at TF32 (TF32_TF32_F32) and 4641 ms at IEEE float32 (F32_F32_F32), against AdamW’s 495 ms, its loss
at each of 30 steps within 0.07 of the float32 iteration’s (0.02 on average). Accumulated, its update’s
microbatch takes 860 ms against 516 and the others about AdamW’s: an update of 65,536 tokens 9.4% over
AdamW’s and one of 262,144 2.5% (dew.optim.muon; ledger 2685, 2752).
orthogonalized
Section titled “orthogonalized”def orthogonalized( updates: optax.Updates, numbers, coeffs: jax.Array, steps: int, eps: float, preconditioning: Preconditioning = 'frobenius', algorithm: jax.lax.DotAlgorithmPreset | None = None, widths=None,) -> optax.UpdatesEach update numbers makes a stack of matrices of (as_matrices) orthogonalized by optax’s
Newton-Schulz (orthogonalized_leaf) in the run’s numerics or algorithm (ns_operands), as Moonlight
distributes
it (arXiv 2502.16982, 3): the matrices of one shape gathered into one stack, spread across the mesh’s
devices whole, each device iterating its own share, and handed back as the updates lie. A matrix split
across devices would exchange partial products at each of the iteration’s products; spread whole, a
step moves each matrix twice, there and back. On a mesh with Explicit axes the spreading runs with
them Auto (jax.sharding.auto_axes), where a constraint may name them and a split stack slices,
each update leaving where it came; on one device, or none in context, each stack iterates whole. A
tree of head widths (head_widths) splits an update’s output side into heads, each one matrix.
orthogonalized_leaf
Section titled “orthogonalized_leaf”def orthogonalized_leaf( update: jax.Array, steps: int, dimension_numbers: optax.contrib.MuonDimensionNumbers | None, *, coeffs=NEWTON_SCHULZ, eps: float = 1e-08, preconditioning: Preconditioning = 'frobenius',) -> jax.Arrayupdate orthogonalized by Muon’s steps quintic Newton-Schulz iterations at coeffs, one triple
or a schedule of them, after preconditioning, optax’s own: its Muon transform with no momentum
(scale_by_muon, beta 0 and no Nesterov step) hands back the update it is given orthogonalized, in
its own dtype, bit for bit its orthogonalize_via_newton_schulz, which optax 0.2.8 does not export.
A leaf with no dimension_numbers is one matrix, its rows the reduction axis (a Linen kernel’s), its
other axes a stack of such matrices.
polar_express
Section titled “polar_express”def polar_express( steps: int, safety: float = 1.01,) -> tuple[tuple[float, float, float], ...]Polar Express’s coefficients for steps iterations as its code applies them (arXiv 2505.16932,
Appendix A): each but the last triple’s polynomial taken at x / safety, (a / s, b / s^3, c / s^5),
the first’s again for its input’s own scaling, X / (safety * ||X||_F), which optax’s iteration, over
X / ||X||_F, then reproduces; the table’s first steps, its last repeated past it.