Skip to content

dew.optim

The optimizer a recipe builds from an OptimConfig.

Every recipe builds the same solver: the config’s learning-rate schedule when it names one, weight decay passed in the optimizer’s own keyword arguments, and global-norm clipping. That is library behavior, so the pieces are here and the recipes call them. The Trainer pools and normalizes the gradient over the whole accumulation window before it calls this solver.

The ‘muon’ entry uses the production parameter-group split that the labs converged on (docs/research/frontier-training.md). AdamW updates the embeddings, the head, the router and the norms, and Muon updates the matrices.

optax.contrib.muon combines the two optimizers, applying each to its own group of parameters. Dew supplies the parameter spec, which says which group a parameter belongs to and which of its axes form the matrix.

NameSummary
NEWTON_SCHULZMuon’s quintic Newton-Schulz coefficients, Jordan’s (modded-nanogpt), optax’s default. Documented in dew.newton_schulz.
BoundsStateThe clamp a bounded param group leaves for the stepped weights (stepped, _keep_within): the updates the group has taken part in, steps, and each of its leaves’ [lower, upper] in the leaf’s dtype, by its path.
CosineA linear warmup from init_value to peak_value, then a cosine decay to end_value at step decay_steps, or at the run’s end when that is None.
DeltaStatedelta_gradient_descent’s state: the updates made so far, which a scheduled rate reads.
ExponentialA geometric decay from offset + init_value to offset + end_value over decay_steps steps (the run’s when None), then constant.
Linearlm-engine’s linear schedule (linear_schedule), with a warmup, a constant rate and a linear decay.
M3GivenStatescale_by_m3’s state where its caller gave its matrices’ dimension numbers, which may name a stored layer’s axes or a stacked leaf’s as written: a stage-local run keeps every layer whole on every stage where those numbers read across a stacked leaf’s layers, as its update’s dataflow says (dew.training.stages.whole_reading), where M3’s own default reads each stored layer.
M3Statescale_by_m3’s state: the updates made so far, the fast momentum M1, the second moment V, the slow momentum M2, the gradients of the chunk in progress summed, and M2 orthogonalized as the chunk began, O2.
OneCycletorch’s OneCycleLR with its default two phases and cosine annealing.
OptimConfigHolds the optimizer, learning-rate schedule and gradient clipping.
ParamGroupA group of parameters the optimizer updates with their own learning rate, momentum, weight decay and bounds.
Powerlm-engine’s power schedule (power_schedule), a power law of the step after a linear warmup.
PowerProfilesStateThe optimizer state of power_profiles, kept beside the wrapped solver’s state.
PowerTailThe linear tail of a power schedule, from the power law’s rate at step start to end_value at step steps, or at the run’s end when that is None.
QKClipStateThe rescale QK-Clip leaves for the stepped weights (stepped): the updates the clip has taken part in, steps, and each query and key leaf’s factor a column of its last axis by its path, every factor one before the first update or where no maxima were sown, so the state keeps one structure.
ScheduleBaseThe base of the schedule records, each of which holds its own fields and builds an optax schedule.
ScheduleSpec
delta_gradient_descentDelta gradient descent (Behrouz et al., Nested Learning, arXiv 2512.24695, eqs.
lapropDreamerV3’s optimizer (dreamerv3/agent.py:342-379): the update clipped adaptively to agc of each parameter’s norm (clip_by_agc), LaProp’s direction of it (scale_by_laprop, momentum b1, mean square b2, eps), stepped at learning_rate. Documented in dew.laprop.
scale_by_m3Multi-scale Momentum Muon, M3 (Behrouz et al., Nested Learning, arXiv 2512.24695, Algorithm 1): the direction (O1 + alpha O2) / sqrt(V + eps), which optax.scale_by_learning_rate steps by.
scale_by_trust_ratiooptax’s scale_by_trust_ratio, LAMB’s (You et al., arXiv 1904.00962), each stored layer’s own: a stage-local run’s leaf takes one ratio a layer, of that layer’s norms (layers_of), and any other leaf its own, as optax’s does.
steppedparams moved by updates, then each post-step effect of the update that took the optimizer’s state from before to after, in the chain’s order: QK-Clip’s rescale (QKClipState), one rounding of gamma * (W + update), and a bounded group’s clamp (BoundsState), exact.

class source

class BoundsState(NamedTuple)

The clamp a bounded param group leaves for the stepped weights (stepped, _keep_within): the updates the group has taken part in, steps, and each of its leaves’ [lower, upper] in the leaf’s dtype, by its path.

dataclass source

class Cosine(
peak_value: float,
warmup_steps: int | str,
end_value: float = 0.0,
init_value: float = 0.0,
decay_steps: int | None = None,
*,
every: int = 1,
)

A linear warmup from init_value to peak_value, then a cosine decay to end_value at step decay_steps, or at the run’s end when that is None.

It is optax.warmup_cosine_decay_schedule. warmup_steps must be smaller than the decay horizon in schedule steps.

warmup_steps: int | str

Its warmup in its own steps, or a share of the run such as 5% (ScheduleBase._warmup): stated, as optax.warmup_cosine_decay_schedule takes it, which has no default either.

def warmup(steps: int | None) -> int
def run_relative() -> bool
def written_warmup() -> int | str | float | None
def marks() -> dict[str, int]
def progressed(
rates: Rates,
progress: jax.Array,
updates: jax.Array,
marks: Mapping[str, jax.Array],
) -> jax.Array

Its warmup, then its cosine decay from where the warmup ended to the run’s end.

def values(steps: int | None, rates: Rates | None = None) -> optax.Schedule

class source

class DeltaState(NamedTuple)

delta_gradient_descent’s state: the updates made so far, which a scheduled rate reads.

dataclass source

class Exponential(
init_value: float,
end_value: float,
decay_steps: int | None = None,
offset: float = 0.0,
*,
every: int = 1,
)

A geometric decay from offset + init_value to offset + end_value over decay_steps steps (the run’s when None), then constant.

Up to decay_steps, the value at step t is offset + init_value * (end_value / init_value) ** (t / decay_steps). That is torch’s ExponentialLR with gamma = (end_value / init_value) ** (1 / decay_steps), stepped decay_steps times and then no more, plus offset. It is optax.exponential_decay with end_value as its bound. SNN-delays anneals DCLS’s kernel width this way, the raw width decaying and a constant added to it.

def peak() -> float

Its higher end, offset added.

def run_relative() -> bool
def progressed(
rates: Rates,
progress: jax.Array,
updates: jax.Array,
marks: Mapping[str, jax.Array],
) -> jax.Array

Its geometric decay over the run’s progress: init_value * (end_value / init_value) ** progress, plus offset.

def values(steps: int | None, rates: Rates | None = None) -> optax.Schedule

dataclass source

class Linear(
peak_value: float,
warmup_steps: int | str = 0,
decay_start: int | None = None,
decay_steps: int | None = None,
end_value: float = 0.0,
*,
every: int = 1,
)

lm-engine’s linear schedule (linear_schedule), with a warmup, a constant rate and a linear decay.

The rate warms up from zero to peak_value, stays constant until decay_start (the end of the warmup when None), then falls linearly to end_value at decay_steps (the run’s end when None).

warmup_steps: int | str

Its warmup in its own steps, or a share of the run such as 5% (ScheduleBase._warmup).

def warmup(steps: int | None) -> int
def run_relative() -> bool
def written_warmup() -> int | str | float | None
def marks() -> dict[str, int]
def progressed(
rates: Rates,
progress: jax.Array,
updates: jax.Array,
marks: Mapping[str, jax.Array],
) -> jax.Array

Its warmup, its constant rate to decay_start (the warmup’s end where None), and its linear decay from there to the run’s end.

def values(steps: int | None, rates: Rates | None = None) -> optax.Schedule

class source

class M3GivenState(M3State)

scale_by_m3’s state where its caller gave its matrices’ dimension numbers, which may name a stored layer’s axes or a stacked leaf’s as written: a stage-local run keeps every layer whole on every stage where those numbers read across a stacked leaf’s layers, as its update’s dataflow says (dew.training.stages.whole_reading), where M3’s own default reads each stored layer.

class source

class M3State(NamedTuple)

scale_by_m3’s state: the updates made so far, the fast momentum M1, the second moment V, the slow momentum M2, the gradients of the chunk in progress summed, and M2 orthogonalized as the chunk began, O2.

dataclass source

class OneCycle(
peak_value: float,
init_value: float | None = None,
end_value: float | None = None,
warmup_fraction: float | str = 0.3,
*,
every: int = 1,
)

torch’s OneCycleLR with its default two phases and cosine annealing.

The value follows a half cosine from init_value to peak_value at step warmup_fraction * steps - 1, then another to end_value at the run’s last step, steps - 1, and holds end_value after it, where torch raises. init_value is torch’s max_lr / div_factor (peak_value / 25 when None) and end_value its init_value / final_div_factor (init_value / 1e4 when None). Adam’s momentum cycle under cycle_momentum=True is OneCycle(peak_value=0.85, init_value=0.95, end_value=0.95) as a group’s b1.

optax’s cosine_onecycle_schedule places its phase boundaries at pct_start * steps and steps, one step later than torch’s (torch/optim/lr_scheduler.py, OneCycleLR.__init__), so its rate peaks one step late and it cannot reproduce a torch run.

warmup_fraction: float | str

The share of the run its rise lasts, torch’s pct_start: a fraction, or a share such as 30%.

def rates() -> dict[str, float]

The peak, and the init and end values with torch’s derived defaults resolved.

def warmup(steps: int | None) -> int

The steps of its rise, to the whole step its rate peaks at (_rise).

def run_relative() -> bool
def written_warmup() -> int | str | float | None
def progressed(
rates: Rates,
progress: jax.Array,
updates: jax.Array,
marks: Mapping[str, jax.Array],
) -> jax.Array

Its rise over warmup_fraction of the run’s progress, then its fall to the run’s end, each a half cosine as over a run of steps; torch’s peak a step before the share’s end and its last value a step before the run’s are the share’s and the run’s ends here.

def values(steps: int | None, rates: Rates | None = None) -> optax.Schedule

dataclass source

class OptimConfig(
optimizer: Optimizer | str = 'adamw',
optimizer_opts: JsonDict = dict(),
learning_rate: Annotated[float | ScheduleSpec | None, _rate_argument()] = None,
weight_decay: float | None = None,
spared: bool = True,
b1: float | None = None,
b2: float | None = None,
param_groups: Annotated[tuple[ParamGroup, ...], record_argument(tuple[ParamGroup, ...])] = (),
clip_grads: float = 0.0,
state_dtype: Literal['float32', 'bfloat16', 'int8'] = 'float32',
forced_weight_normalization: bool = False,
ema_profiles: tuple[float, ...] = (),
)

Holds the optimizer, learning-rate schedule and gradient clipping.

optimizer: Optimizer | str

One of Dew’s optimizers (OPTIMIZER_MAP), or any Optax optimizer factory by its import path, optax:sgd or optax:adafactor (optimizers), built as factory(learning_rate, **optimizer_opts) with b1, b2 and weight_decay where the config sets them, so an experiment on an Optax transformation records the same optimizer it ran. The decay spares biases and norm scales for a factory that takes a mask.

learning_rate: Annotated[float | ScheduleSpec | None, _rate_argument()]

The learning rate: a constant, or a schedule as one typed record per kind (dew.optim); unset, the optimizer’s own (OWN_RATES: Gauss-Newton’s whole step, 1.0, the rest’s 2.7e-4), which the config then records.

The kinds are cosine, power (lm-engine’s power law with an optional linear tail), linear, one_cycle (torch’s OneCycleLR) and exponential; each record holds only its own fields, and every steps any of them once per that many updates. On the command line it is one value: --optim.learning-rate 3e-4, or a schedule’s record, --optim.learning-rate '{"class": "cosine", "fields": {...}}'.

weight_decay: float | None

The weight decay passed to the optimizer, None keeping its own (adamw’s 1e-4, the others’ 0); for ‘adam’ it is torch’s coupled L2 penalty, added to the gradient before the moments read it, for ‘adamw’ the decoupled decay. Either decay, the default included, spares biases and vectors such as a norm’s scale, timm’s split for every optimizer and for an Optax factory given one (dew.optim.weight_decay_mask, its mask), unless spared is off; Muon decays them as Moonlight does, and a param group decays all it claims unless it says (ParamGroup.spared).

spared: bool

Whether the decay spares biases and vectors such as a norm’s scale (weight_decay_mask), where the optimizer takes a mask: nanoGPT’s, Megatron Core’s and timm’s split, since decaying a norm’s scale or a bias toward zero regularizes nothing a matrix’s decay does and shifts what the layer computes. Optax decays every parameter (mask=None), which False restores. A param group decays all it claims unless its own spared says otherwise (ParamGroup.spared); a run’s default groups follow this.

b1: float | None

First-moment decay, for adam, adamw and lamb and laprop’s momentum; None keeps the optimizer’s.

b2: float | None

Second-moment decay, for adam, adamw and lamb and laprop’s mean square; None keeps the optimizer’s.

param_groups: Annotated[tuple[ParamGroup, ...], record_argument(tuple[ParamGroup, ...])]

Per-group learning rates, momentum schedules, weight decay and bounds; the first matching group wins.

Empty runs one optimizer over every parameter. ParamGroup.mup is lm-engine’s muP split.

state_dtype: Literal['float32', 'bfloat16', 'int8']

The dtype of Adam’s moments in memory.

float32 stores adam’s and adamw’s two moments, and Muon’s momentum, in float32 whatever the parameters’ dtype, so bfloat16 or float16 weights train on full-precision moments. bfloat16 stores both with stochastic rounding (dew._moments.bf16_moments), for adam and adamw only. That halves the optimizer state and reduces the update’s memory traffic. int8 stores both blockwise in 8 bits (dew._moments.int8_moments, bitsandbytes’ and torchao’s 8-bit Adam), for adam and adamw only: a quarter of float32’s, about 2 bytes a parameter for the two.

forced_weight_normalization: bool

Whether to renormalize every magnitude-preserving weight (dew.nn.mp.MPConv) after each update.

This is EDM2’s forced weight normalization, which its edm2_unet trains with (dew.nn.mp.forced_weight_normalization).

ema_profiles: tuple[float, ...]

Relative standard deviations of the power-function EMAs a run keeps for post-hoc EMA.

See dew.optim.power_profiles; Karras et al. use (0.05, 0.10). Every checkpoint save snapshots these averages, and Checkpoints.posthoc_ema builds an average with any other relative standard deviation from the snapshots. Empty keeps none.

rate: float | ScheduleBase

Its learning rate, the optimizer’s own where it set none (__post_init__).

def schedules() -> list[tuple[str, ScheduleBase]]

The schedules its solvers run, by where the config holds them: its learning rate’s, or each param group’s rate (ParamGroup.rate, the config’s own where the group names none) and b1.

def reads_progress() -> bool

Whether a run that counts no updates ahead (one bounded in examples) hands its optimizer its progress: a schedule of its whose horizon is the run’s (ScheduleBase.run_relative).

def warmups(steps: int | None, clocked: bool = False) -> dict[str, int | str | float]

Each of its schedules’ warmup in that schedule’s own steps over a run of steps updates, its share of the run resolved (ScheduleBase.warmup), by where the config holds it, as a run’s plan records it; a schedule with no warmup of steps is not among them. A clocked run counts no updates ahead, so a warmup over the run’s progress is recorded as written, its share of the examples the run consumes (ScheduleBase.progressed).

def groups(params) -> dict[str, str] | None

Where this optimizer’s split sends each of params and why (muon_partition), for the optimizers that split by Muon’s rule (‘muon’, ‘muonclip’); None for one that splits none. Read under the axes the modules declare, as its state is initialized.

def build(steps: int | None, clocked: bool = False) -> optax.GradientTransformation

Build the optimizer this config describes, with its schedule, parameter groups and clipping.

steps is the run’s length in updates, which the schedule decays over unless the config names its own end; a clocked run counts none ahead (it is bounded in examples), and a schedule whose horizon is the run’s reads its progress, which each update is handed (scheduled_solver). With param_groups, one optimizer runs per group under optax.multi_transform, each on its own schedule (the config’s when it names none) times its multiplier, with its own weight decay, b1 schedule and bounds (ParamGroup.solver); the global-norm clip still reads every gradient together, before the groups split them. Each solver holds its rate’s fields in its state (dew.optim.scheduled_solver), so runs at rates of one schedule’s timings compile one step.

def fresh_groups() -> tuple[ParamGroup, ...]

The groups a run takes where it names none and draws modules fresh beside a loaded base: those modules at FRESH_LEARNING_RATE at the peak of this config’s schedule, on its shape (its rate times their quotient), or at its own where that is higher, and every other parameter at its own, each sparing what the ungrouped decay spares (ParamGroup.spared).

def with_fresh(modules: Sequence[str]) -> OptimConfig

This config with each group that claims the fresh modules (ParamGroup.fresh) matching the parameters of modules, each a module’s path under params as a parameter’s name begins (parameter_name), ahead of its own patterns: the groups a run records and trains in, its modules drawn fresh beside a loaded base (Objective.fresh_modules). Refused where such a group then matches nothing, in a run that draws nothing fresh.

def rates() -> dict[str | None, dict[str, float]]

The rate fields each of build’s solvers holds in its state (dew.optim.scheduled_solver), by param group, or under None for the one solver of a config without groups. A resumed run sets them over its checkpoint’s (with_rates), so the configured rate is the one it steps at.

dataclass source

class ParamGroup(
name: str,
patterns: tuple[str, ...] = (),
learning_rate_multiplier: float = 1.0,
weight_decay: float | None = None,
spared: bool = False,
schedule: ScheduleBase | None = None,
b1: ScheduleBase | None = None,
bounds: tuple[float, float] | None = None,
fresh: bool = False,
)

A group of parameters the optimizer updates with their own learning rate, momentum, weight decay and bounds.

patterns are fnmatch patterns matched against a parameter’s path, which is its dict keys joined by ’/’ (layers_3/self_attn/q_proj/kernel); * also matches ’/’. A parameter belongs to the first group in OptimConfig.param_groups that has a pattern it matches, as in lm-engine, and a parameter that matches no group raises ValueError.

The group’s learning rate is its own schedule (the config’s learning_rate when None) times learning_rate_multiplier. b1 schedules the first-moment decay of adam, adamw and lamb, as torch’s OneCycleLR(cycle_momentum=True) cycles Adam’s beta1; None keeps the optimizer’s own. weight_decay replaces the config’s weight decay; None keeps it. For ‘adam’ that decay is torch’s coupled L2 (_coupled_decay), and for ‘adamw’ the decoupled one. bounds clamps each parameter into [lower, upper] after every update (_keep_within).

spared: bool

Whether the group’s decay spares its biases and vectors (weight_decay_mask). A group decays every parameter it claims unless it does, as a torch optimizer’s parameter group does, so a run’s groups mirror the reference’s own groups (lm_engine’s, a torch recipe’s); the split an ungrouped optimizer takes (OptimConfig.spared) is the groups’ only where they say so.

fresh: bool

Whether the group claims the modules its run draws fresh beside a loaded base, ahead of its own patterns (Objective.fresh_modules): what OptimConfig.with_fresh names before a run records the group.

def rates(learning_rate: float | ScheduleBase) -> dict[str, float]

This group’s rate-valued fields (RateState): its own schedule’s, else the config’s learning_rate’s, times learning_rate_multiplier.

def rate(learning_rate: R) -> R | ScheduleBase

The rate its solver runs at: its own schedule, or the config’s learning_rate where it names none.

def solver(
make: Callable[..., optax.GradientTransformation],
learning_rate: float | ScheduleBase,
steps: int | None,
opts: Mapping[str, object],
*,
clocked: bool = False,
) -> optax.GradientTransformation

Build this group’s optimizer: make at the group’s rate, its fields held in state (scheduled_solver), with the group’s b1 schedule and bounds, over a run of steps updates, or of the progress of a clocked one.

A b1 schedule replaces any b1 in opts, and needs a make that names a b1 argument (adam, adamw, lamb; not muon).

def mup(width_multiplier: float) -> tuple[ParamGroup, ...]

Return lm-engine’s muP parameter groups, in lm-engine’s order.

The groups are: norms, biases and dt_bias at the base rate without weight decay; the token embeddings at the base rate with weight decay; and everything else, including the router, A_log, D and the conv taps, at the base rate divided by width_multiplier. width_multiplier is lm-engine’s m_width, which is the model’s logits_scaling.

dataclass source

class Power(
peak_value: float,
warmup_steps: int | str,
a: float,
b: float = -0.51,
c: float = 1.0,
tail: PowerTail | None = None,
*,
every: int = 1,
)

lm-engine’s power schedule (power_schedule), a power law of the step after a linear warmup.

After the warmup, the rate is min(peak_value, a * (step * c) ** b). With a tail, a linear decay follows the power law, as in the last 29% of Rigel’s run. lm-engine’s examples set a to 4 * batch size and c to the tokens per step.

warmup_steps: int | str

Its warmup in its own steps, or a share of the run such as 5% (ScheduleBase._warmup).

def warmup(steps: int | None) -> int
def peak() -> float

peak_value, the cap the power law climbs to.

def run_relative() -> bool
def written_warmup() -> int | str | float | None
def marks() -> dict[str, int]
def progressed(
rates: Rates,
progress: jax.Array,
updates: jax.Array,
marks: Mapping[str, jax.Array],
) -> jax.Array

Its power law of its updates after its warmup, and its tail from tail.start to the run’s end. A warmup written as a share of the run rises over that share of the progress to the law’s value at the update it reaches, the law scaled by how far through it the run stands.

def values(steps: int | None, rates: Rates | None = None) -> optax.Schedule

class source

class PowerProfilesState(NamedTuple)

The optimizer state of power_profiles, kept beside the wrapped solver’s state.

updates: jax.Array

The number of updates made so far, the t in the averages’ power-function profiles.

stds: jax.Array

The relative standard deviation of each average.

powers: jax.Array

Each average’s γ + 1, the power its decay raises (1 - 1/t) to, solved from its std when the state starts: a resumed average decays as it was tracked, whatever stds the resuming run configures.

dataclass source

class PowerTail(start: int, steps: int | None = None, end_value: float = 0.0)

The linear tail of a power schedule, from the power law’s rate at step start to end_value at step steps, or at the run’s end when that is None.

class source

class QKClipState(NamedTuple)

The rescale QK-Clip leaves for the stepped weights (stepped): the updates the clip has taken part in, steps, and each query and key leaf’s factor a column of its last axis by its path, every factor one before the first update or where no maxima were sown, so the state keeps one structure. A stage-local run’s leaf holds one entry a stored layer, by the layer’s path (layers_of), as an unstacked tree holds it.

dataclass source

class ScheduleBase(*, every: int = 1)

The base of the schedule records, each of which holds its own fields and builds an optax schedule.

A run’s record names the schedule’s class and holds only the fields that schedule reads; dew.registry.schedules holds the short aliases. A subclass builds its values in values; schedule reads them. The fields it marks metadata={'rate': True} are its rates (rates), which a solver holds as operands (scheduled_solver) and values reads from rates, so a run’s program does not depend on them; a subclass that marks none implements values(steps) alone.

every: int

The updates each of the schedule’s own steps lasts.

torch recipes step a scheduler once an epoch, and with every set to the updates of an epoch this one holds each value through an epoch as torch’s does. A run of steps updates is then steps // every of the schedule’s own steps, and every field that counts steps (warmup_steps, decay_steps) counts those.

def rates() -> dict[str, float]

Each rate-valued field’s value (metadata={'rate': True}) by its dotted path, nested records such as PowerTail’s too, a None field’s derived default resolved where the schedule derives one (OneCycle’s) and left out otherwise.

def schedule(steps: int | None, rates: Rates | None = None) -> optax.Schedule

Return the value at each update of a run of steps updates, as an optax schedule, its rate-valued fields rates’ (rates() by default): operands in a solver’s state, or the configured values.

def warmup(steps: int | None) -> int | None

Its warmup in its own steps over a run of steps updates, its share of the run resolved (_warmup), as a run’s plan records it; None for a schedule with no warmup of steps.

def values(steps: int | None, rates: Rates | None = None) -> optax.Schedule

Return the value at each of the schedule’s own steps, over steps of them, from its rate-valued fields rates (rates()). A value derived from the rates alone, as a ratio of two, is the configured fields’, which a solver’s operands keep: they differ from them by one factor, a param group’s multiplier.

def peak() -> float

The highest value the schedule takes: its largest rate-valued field, a warmup’s or decay’s ends, where its formula reaches no higher.

def run_relative() -> bool

Whether its horizon is the run’s, so a run that counts no updates ahead (one bounded in examples) reads it over the run’s progress (progressed); a schedule whose every horizon is its own reads its updates there as anywhere.

def written_warmup() -> int | str | float | None

Its warmup as written, in its own steps or as a share of the run, which a run that counts no updates ahead records as it is (OptimConfig.warmups); None where it has none.

def marks() -> dict[str, int]

The update counts its law on a run’s progress starts a piece at (progressed), by name: the end of a warmup counted in updates, say. The run records where its progress stood as its updates reached each (ProgressState.marks).

def progressed(
rates: Rates,
progress: jax.Array,
updates: jax.Array,
marks: Mapping[str, jax.Array],
) -> jax.Array

Its value where a run that counts no updates ahead stands: at progress, the share of the run’s bound it has consumed (dew.state.TrainState.consumed), after updates updates, the progress it stood at as it reached each count it marks. Each piece written in updates (a warmup of 1000) reads the updates, and each that lasts to the run’s end reads the progress from where it starts, its law kept: a share of the run (5%) is that share of the progress, and the run’s end its whole. A schedule’s own steps (every) step its updates alone.

attribute source

function source

def delta_gradient_descent(
learning_rate: float | optax.Schedule,
) -> optax.GradientTransformationExtraArgs

Delta gradient descent (Behrouz et al., Nested Learning, arXiv 2512.24695, eqs. 56-57): each matrix steps to the minimizer of |W x - u|**2 / 2 + |W - W_t|**2 / (2 eta), where x is the step’s input and u = -dL/dy its output’s negative gradient, so the step forgets what W_t held along x.

By Sherman-Morrison the minimizer is W_t (I - eta' x x^T) - eta' dL/dW with eta' = eta / (1 + eta |x|**2), the paper’s eta / (1 + eta) at its normalized |x| = 1. A kernel holds its input on axis 0, as a Linen kernel does (y = x K), so its step is -eta' (x (x^T K) + dL/dK) over its other axes. The update takes inputs, a tree of x per leaf, the parameters’ structure with None where a leaf has none: that leaf steps by gradient descent, the rule at x = 0. An update given no inputs, or no parameters, is refused rather than run as plain gradient descent.

function source

def scale_by_m3(
*,
beta1: float,
beta2: float,
beta3: float,
alpha: float,
eps: float,
frequency: int,
ns_steps: int = 5,
weight_dimension_numbers=None,
ns_precision: jax.lax.DotAlgorithmPreset | str | None = None,
) -> optax.GradientTransformation

Multi-scale Momentum Muon, M3 (Behrouz et al., Nested Learning, arXiv 2512.24695, Algorithm 1): the direction (O1 + alpha O2) / sqrt(V + eps), which optax.scale_by_learning_rate steps by.

Each update adds beta1 g to the fast momentum M1 and beta2 g**2 to the second moment V, the algorithm’s sums, neither decaying, and orthogonalizes M1 by ns_steps Newton-Schulz iterations, Muon’s in the run’s numerics or at ns_precision’s dot algorithm, each matrix whole on one device and the matrices spread over the mesh (orthogonalized), each leaf shaped by its weight_dimension_numbers, a tree or a function of the updates as Muon’s are; a leaf with none is one matrix, its rows the reduction. Its state is kept in float32 at least (at_least_fp32), so a bfloat16 matrix’s momenta and gradients sum without rounding to its width, and the directions come back in that dtype. Once every frequency updates the slow momentum M2 adds beta3 times the chunk’s gradients and is orthogonalized, O2, which the next chunk’s updates read: the first chunk’s is zero. The algorithm’s sum, over i = (k-1) f .. k f, and eq. 75’s, over i = t - C .. t, each hold f + 1 gradients, the one at a chunk’s boundary in two chunks; here each enters M2 once, with its own chunk’s. The paper states no values for its factors, so each is the caller’s.

function source

def scale_by_trust_ratio(
min_norm: float = 0.0,
trust_coefficient: float = 1.0,
eps: float = 0.0,
) -> optax.GradientTransformation

optax’s scale_by_trust_ratio, LAMB’s (You et al., arXiv 1904.00962), each stored layer’s own: a stage-local run’s leaf takes one ratio a layer, of that layer’s norms (layers_of), and any other leaf its own, as optax’s does.

function source

def stepped(params, updates, before, after)

params moved by updates, then each post-step effect of the update that took the optimizer’s state from before to after, in the chain’s order: QK-Clip’s rescale (QKClipState), one rounding of gamma * (W + update), and a bounded group’s clamp (BoundsState), exact. The weights a step stores, and the only way to form them (the trainer’s transaction, power_profiles’ averages). An effect whose transform the update did not run, a group a mask left out or an update a wrapper deferred or rejected (optax.MultiSteps, optax.apply_if_finite), whose state comes back as it was, leaves the weights as the update moved them.