dew.optim
The optimizer a recipe builds from an OptimConfig.
Every recipe builds the same solver: the config’s learning-rate schedule when it names one, weight decay passed in the optimizer’s own keyword arguments, and global-norm clipping. That is library behavior, so the pieces are here and the recipes call them. The Trainer pools and normalizes the gradient over the whole accumulation window before it calls this solver.
The ‘muon’ entry uses the production parameter-group split that the labs converged on (docs/research/frontier-training.md). AdamW updates the embeddings, the head, the router and the norms, and Muon updates the matrices.
optax.contrib.muon combines the two optimizers, applying each to its own
group of parameters. Dew supplies the parameter spec, which says which group
a parameter belongs to and which of its axes form the matrix.
| Name | Summary |
|---|---|
NEWTON_SCHULZ | Muon’s quintic Newton-Schulz coefficients, Jordan’s (modded-nanogpt), optax’s default. Documented in dew.newton_schulz. |
BoundsState | The clamp a bounded param group leaves for the stepped weights (stepped, _keep_within): the updates the group has taken part in, steps, and each of its leaves’ [lower, upper] in the leaf’s dtype, by its path. |
Cosine | A linear warmup from init_value to peak_value, then a cosine decay to end_value at step decay_steps, or at the run’s end when that is None. |
DeltaState | delta_gradient_descent’s state: the updates made so far, which a scheduled rate reads. |
Exponential | A geometric decay from offset + init_value to offset + end_value over decay_steps steps (the run’s when None), then constant. |
Linear | lm-engine’s linear schedule (linear_schedule), with a warmup, a constant rate and a linear decay. |
M3GivenState | scale_by_m3’s state where its caller gave its matrices’ dimension numbers, which may name a stored layer’s axes or a stacked leaf’s as written: a stage-local run keeps every layer whole on every stage where those numbers read across a stacked leaf’s layers, as its update’s dataflow says (dew.training.stages.whole_reading), where M3’s own default reads each stored layer. |
M3State | scale_by_m3’s state: the updates made so far, the fast momentum M1, the second moment V, the slow momentum M2, the gradients of the chunk in progress summed, and M2 orthogonalized as the chunk began, O2. |
OneCycle | torch’s OneCycleLR with its default two phases and cosine annealing. |
OptimConfig | Holds the optimizer, learning-rate schedule and gradient clipping. |
ParamGroup | A group of parameters the optimizer updates with their own learning rate, momentum, weight decay and bounds. |
Power | lm-engine’s power schedule (power_schedule), a power law of the step after a linear warmup. |
PowerProfilesState | The optimizer state of power_profiles, kept beside the wrapped solver’s state. |
PowerTail | The linear tail of a power schedule, from the power law’s rate at step start to end_value at step steps, or at the run’s end when that is None. |
QKClipState | The rescale QK-Clip leaves for the stepped weights (stepped): the updates the clip has taken part in, steps, and each query and key leaf’s factor a column of its last axis by its path, every factor one before the first update or where no maxima were sown, so the state keeps one structure. |
ScheduleBase | The base of the schedule records, each of which holds its own fields and builds an optax schedule. |
ScheduleSpec | |
delta_gradient_descent | Delta gradient descent (Behrouz et al., Nested Learning, arXiv 2512.24695, eqs. |
laprop | DreamerV3’s optimizer (dreamerv3/agent.py:342-379): the update clipped adaptively to agc of each parameter’s norm (clip_by_agc), LaProp’s direction of it (scale_by_laprop, momentum b1, mean square b2, eps), stepped at learning_rate. Documented in dew.laprop. |
scale_by_m3 | Multi-scale Momentum Muon, M3 (Behrouz et al., Nested Learning, arXiv 2512.24695, Algorithm 1): the direction (O1 + alpha O2) / sqrt(V + eps), which optax.scale_by_learning_rate steps by. |
scale_by_trust_ratio | optax’s scale_by_trust_ratio, LAMB’s (You et al., arXiv 1904.00962), each stored layer’s own: a stage-local run’s leaf takes one ratio a layer, of that layer’s norms (layers_of), and any other leaf its own, as optax’s does. |
stepped | params moved by updates, then each post-step effect of the update that took the optimizer’s state from before to after, in the chain’s order: QK-Clip’s rescale (QKClipState), one rounding of gamma * (W + update), and a bounded group’s clamp (BoundsState), exact. |
BoundsState
Section titled “BoundsState”class BoundsState(NamedTuple)The clamp a bounded param group leaves for the stepped weights (stepped, _keep_within): the
updates the group has taken part in, steps, and each of its leaves’ [lower, upper] in the
leaf’s dtype, by its path.
Cosine
Section titled “Cosine”class Cosine( peak_value: float, warmup_steps: int | str, end_value: float = 0.0, init_value: float = 0.0, decay_steps: int | None = None, *, every: int = 1,)A linear warmup from init_value to peak_value, then a cosine decay to end_value at
step decay_steps, or at the run’s end when that is None.
It is optax.warmup_cosine_decay_schedule.
warmup_steps must be smaller than the decay horizon in schedule steps.
warmup_steps: int | str-
Its warmup in its own steps, or a share of the run such as
5%(ScheduleBase._warmup): stated, asoptax.warmup_cosine_decay_scheduletakes it, which has no default either.
Cosine.warmup
Section titled “Cosine.warmup”def warmup(steps: int | None) -> intCosine.run_relative
Section titled “Cosine.run_relative”def run_relative() -> boolCosine.written_warmup
Section titled “Cosine.written_warmup”def written_warmup() -> int | str | float | NoneCosine.marks
Section titled “Cosine.marks”def marks() -> dict[str, int]Cosine.progressed
Section titled “Cosine.progressed”def progressed( rates: Rates, progress: jax.Array, updates: jax.Array, marks: Mapping[str, jax.Array],) -> jax.ArrayIts warmup, then its cosine decay from where the warmup ended to the run’s end.
Cosine.values
Section titled “Cosine.values”def values(steps: int | None, rates: Rates | None = None) -> optax.ScheduleDeltaState
Section titled “DeltaState”class DeltaState(NamedTuple)delta_gradient_descent’s state: the updates made so far, which a scheduled rate reads.
Exponential
Section titled “Exponential”class Exponential( init_value: float, end_value: float, decay_steps: int | None = None, offset: float = 0.0, *, every: int = 1,)A geometric decay from offset + init_value to offset + end_value over
decay_steps steps (the run’s when None), then constant.
Up to decay_steps, the value at step t is
offset + init_value * (end_value / init_value) ** (t / decay_steps). That is torch’s
ExponentialLR with gamma = (end_value / init_value) ** (1 / decay_steps), stepped
decay_steps times and then no more, plus offset. It is
optax.exponential_decay with end_value as its bound. SNN-delays anneals
DCLS’s kernel width this way, the raw width decaying and a constant
added to it.
Exponential.peak
Section titled “Exponential.peak”def peak() -> floatIts higher end, offset added.
Exponential.run_relative
Section titled “Exponential.run_relative”def run_relative() -> boolExponential.progressed
Section titled “Exponential.progressed”def progressed( rates: Rates, progress: jax.Array, updates: jax.Array, marks: Mapping[str, jax.Array],) -> jax.ArrayIts geometric decay over the run’s progress: init_value * (end_value / init_value) ** progress,
plus offset.
Exponential.values
Section titled “Exponential.values”def values(steps: int | None, rates: Rates | None = None) -> optax.ScheduleLinear
Section titled “Linear”class Linear( peak_value: float, warmup_steps: int | str = 0, decay_start: int | None = None, decay_steps: int | None = None, end_value: float = 0.0, *, every: int = 1,)lm-engine’s linear schedule (linear_schedule), with a warmup, a constant rate and a linear decay.
The rate warms up from zero to peak_value, stays constant until
decay_start (the end of the warmup when None), then falls linearly to
end_value at decay_steps (the run’s end when None).
warmup_steps: int | str-
Its warmup in its own steps, or a share of the run such as
5%(ScheduleBase._warmup).
Linear.warmup
Section titled “Linear.warmup”def warmup(steps: int | None) -> intLinear.run_relative
Section titled “Linear.run_relative”def run_relative() -> boolLinear.written_warmup
Section titled “Linear.written_warmup”def written_warmup() -> int | str | float | NoneLinear.marks
Section titled “Linear.marks”def marks() -> dict[str, int]Linear.progressed
Section titled “Linear.progressed”def progressed( rates: Rates, progress: jax.Array, updates: jax.Array, marks: Mapping[str, jax.Array],) -> jax.ArrayIts warmup, its constant rate to decay_start (the warmup’s end where None), and its linear
decay from there to the run’s end.
Linear.values
Section titled “Linear.values”def values(steps: int | None, rates: Rates | None = None) -> optax.ScheduleM3GivenState
Section titled “M3GivenState”class M3GivenState(M3State)scale_by_m3’s state where its caller gave its matrices’ dimension numbers, which may name a stored
layer’s axes or a stacked leaf’s as written: a stage-local run keeps every layer whole on every stage
where those numbers read across a stacked leaf’s layers, as its update’s dataflow says
(dew.training.stages.whole_reading), where M3’s own default reads each stored layer.
M3State
Section titled “M3State”class M3State(NamedTuple)scale_by_m3’s state: the updates made so far, the fast momentum M1, the second moment V, the slow
momentum M2, the gradients of the chunk in progress summed, and M2 orthogonalized as the chunk
began, O2.
OneCycle
Section titled “OneCycle”class OneCycle( peak_value: float, init_value: float | None = None, end_value: float | None = None, warmup_fraction: float | str = 0.3, *, every: int = 1,)torch’s OneCycleLR with its default two phases and cosine annealing.
The value follows a half cosine from init_value to peak_value at step
warmup_fraction * steps - 1, then another to end_value at the run’s last
step, steps - 1, and holds end_value after it, where torch raises. init_value
is torch’s max_lr / div_factor (peak_value / 25 when None) and end_value its
init_value / final_div_factor (init_value / 1e4 when None). Adam’s momentum cycle
under cycle_momentum=True is OneCycle(peak_value=0.85, init_value=0.95, end_value=0.95) as a group’s b1.
optax’s cosine_onecycle_schedule places its phase boundaries at
pct_start * steps and steps, one step later than torch’s
(torch/optim/lr_scheduler.py, OneCycleLR.__init__), so its rate peaks
one step late and it cannot reproduce a torch run.
warmup_fraction: float | str-
The share of the run its rise lasts, torch’s
pct_start: a fraction, or a share such as30%.
OneCycle.rates
Section titled “OneCycle.rates”def rates() -> dict[str, float]The peak, and the init and end values with torch’s derived defaults resolved.
OneCycle.warmup
Section titled “OneCycle.warmup”def warmup(steps: int | None) -> intThe steps of its rise, to the whole step its rate peaks at (_rise).
OneCycle.run_relative
Section titled “OneCycle.run_relative”def run_relative() -> boolOneCycle.written_warmup
Section titled “OneCycle.written_warmup”def written_warmup() -> int | str | float | NoneOneCycle.progressed
Section titled “OneCycle.progressed”def progressed( rates: Rates, progress: jax.Array, updates: jax.Array, marks: Mapping[str, jax.Array],) -> jax.ArrayIts rise over warmup_fraction of the run’s progress, then its fall to the run’s end, each a
half cosine as over a run of steps; torch’s peak a step before the share’s end and its last
value a step before the run’s are the share’s and the run’s ends here.
OneCycle.values
Section titled “OneCycle.values”def values(steps: int | None, rates: Rates | None = None) -> optax.ScheduleOptimConfig
Section titled “OptimConfig”class OptimConfig( optimizer: Optimizer | str = 'adamw', optimizer_opts: JsonDict = dict(), learning_rate: Annotated[float | ScheduleSpec | None, _rate_argument()] = None, weight_decay: float | None = None, spared: bool = True, b1: float | None = None, b2: float | None = None, param_groups: Annotated[tuple[ParamGroup, ...], record_argument(tuple[ParamGroup, ...])] = (), clip_grads: float = 0.0, state_dtype: Literal['float32', 'bfloat16', 'int8'] = 'float32', forced_weight_normalization: bool = False, ema_profiles: tuple[float, ...] = (),)Holds the optimizer, learning-rate schedule and gradient clipping.
optimizer: Optimizer | str-
One of Dew’s optimizers (
OPTIMIZER_MAP), or any Optax optimizer factory by its import path,optax:sgdoroptax:adafactor(optimizers), built asfactory(learning_rate, **optimizer_opts)withb1,b2andweight_decaywhere the config sets them, so an experiment on an Optax transformation records the same optimizer it ran. The decay spares biases and norm scales for a factory that takes amask. learning_rate: Annotated[float | ScheduleSpec | None, _rate_argument()]-
The learning rate: a constant, or a schedule as one typed record per kind (
dew.optim); unset, the optimizer’s own (OWN_RATES: Gauss-Newton’s whole step, 1.0, the rest’s 2.7e-4), which the config then records.The kinds are cosine, power (lm-engine’s power law with an optional linear tail), linear, one_cycle (torch’s
OneCycleLR) and exponential; each record holds only its own fields, andeverysteps any of them once per that many updates. On the command line it is one value:--optim.learning-rate 3e-4, or a schedule’s record,--optim.learning-rate '{"class": "cosine", "fields": {...}}'. weight_decay: float | None-
The weight decay passed to the optimizer, None keeping its own (adamw’s 1e-4, the others’ 0); for ‘adam’ it is torch’s coupled L2 penalty, added to the gradient before the moments read it, for ‘adamw’ the decoupled decay. Either decay, the default included, spares biases and vectors such as a norm’s scale, timm’s split for every optimizer and for an Optax factory given one (
dew.optim.weight_decay_mask, itsmask), unlesssparedis off; Muon decays them as Moonlight does, and a param group decays all it claims unless it says (ParamGroup.spared). spared: bool-
Whether the decay spares biases and vectors such as a norm’s scale (
weight_decay_mask), where the optimizer takes a mask: nanoGPT’s, Megatron Core’s and timm’s split, since decaying a norm’s scale or a bias toward zero regularizes nothing a matrix’s decay does and shifts what the layer computes. Optax decays every parameter (mask=None), which False restores. A param group decays all it claims unless its ownsparedsays otherwise (ParamGroup.spared); a run’s default groups follow this. b1: float | None-
First-moment decay, for adam, adamw and lamb and laprop’s momentum; None keeps the optimizer’s.
b2: float | None-
Second-moment decay, for adam, adamw and lamb and laprop’s mean square; None keeps the optimizer’s.
param_groups: Annotated[tuple[ParamGroup, ...], record_argument(tuple[ParamGroup, ...])]-
Per-group learning rates, momentum schedules, weight decay and bounds; the first matching group wins.
Empty runs one optimizer over every parameter.
ParamGroup.mupis lm-engine’s muP split. state_dtype: Literal['float32', 'bfloat16', 'int8']-
The dtype of Adam’s moments in memory.
float32 stores adam’s and adamw’s two moments, and Muon’s momentum, in float32 whatever the parameters’ dtype, so bfloat16 or float16 weights train on full-precision moments. bfloat16 stores both with stochastic rounding (
dew._moments.bf16_moments), for adam and adamw only. That halves the optimizer state and reduces the update’s memory traffic. int8 stores both blockwise in 8 bits (dew._moments.int8_moments, bitsandbytes’ and torchao’s 8-bit Adam), for adam and adamw only: a quarter of float32’s, about 2 bytes a parameter for the two. forced_weight_normalization: bool-
Whether to renormalize every magnitude-preserving weight (
dew.nn.mp.MPConv) after each update.This is EDM2’s forced weight normalization, which its
edm2_unettrains with (dew.nn.mp.forced_weight_normalization). ema_profiles: tuple[float, ...]-
Relative standard deviations of the power-function EMAs a run keeps for post-hoc EMA.
See
dew.optim.power_profiles; Karras et al. use (0.05, 0.10). Every checkpoint save snapshots these averages, andCheckpoints.posthoc_emabuilds an average with any other relative standard deviation from the snapshots. Empty keeps none. rate: float | ScheduleBase-
Its learning rate, the optimizer’s own where it set none (
__post_init__).
OptimConfig.schedules
Section titled “OptimConfig.schedules”def schedules() -> list[tuple[str, ScheduleBase]]The schedules its solvers run, by where the config holds them: its learning rate’s, or each
param group’s rate (ParamGroup.rate, the config’s own where the group names none) and b1.
OptimConfig.reads_progress
Section titled “OptimConfig.reads_progress”def reads_progress() -> boolWhether a run that counts no updates ahead (one bounded in examples) hands its optimizer its
progress: a schedule of its whose horizon is the run’s (ScheduleBase.run_relative).
OptimConfig.warmups
Section titled “OptimConfig.warmups”def warmups(steps: int | None, clocked: bool = False) -> dict[str, int | str | float]Each of its schedules’ warmup in that schedule’s own steps over a run of steps updates,
its share of the run resolved (ScheduleBase.warmup), by where the config holds it, as a
run’s plan records it; a schedule with no warmup of steps is not among them. A clocked run
counts no updates ahead, so a warmup over the run’s progress is recorded as written, its share
of the examples the run consumes (ScheduleBase.progressed).
OptimConfig.groups
Section titled “OptimConfig.groups”def groups(params) -> dict[str, str] | NoneWhere this optimizer’s split sends each of params and why (muon_partition), for the
optimizers that split by Muon’s rule (‘muon’, ‘muonclip’); None for one that splits none. Read
under the axes the modules declare, as its state is initialized.
OptimConfig.build
Section titled “OptimConfig.build”def build(steps: int | None, clocked: bool = False) -> optax.GradientTransformationBuild the optimizer this config describes, with its schedule, parameter groups and clipping.
steps is the run’s length in updates, which the schedule decays over unless the
config names its own end; a clocked run counts none ahead (it is bounded in examples), and a
schedule whose horizon is the run’s reads its progress, which each update is handed
(scheduled_solver). With param_groups, one optimizer runs per group under
optax.multi_transform, each on its own schedule (the config’s when it names
none) times its multiplier, with its own weight decay, b1 schedule and bounds
(ParamGroup.solver); the global-norm clip still reads every gradient
together, before the groups split them. Each solver holds its rate’s
fields in its state (dew.optim.scheduled_solver), so runs at
rates of one schedule’s timings compile one step.
OptimConfig.fresh_groups
Section titled “OptimConfig.fresh_groups”def fresh_groups() -> tuple[ParamGroup, ...]The groups a run takes where it names none and draws modules fresh beside a loaded base: those
modules at FRESH_LEARNING_RATE at the peak of this config’s schedule, on its shape (its rate
times their quotient), or at its own where that is higher, and every other parameter at its own,
each sparing what the ungrouped decay spares (ParamGroup.spared).
OptimConfig.with_fresh
Section titled “OptimConfig.with_fresh”def with_fresh(modules: Sequence[str]) -> OptimConfigThis config with each group that claims the fresh modules (ParamGroup.fresh) matching the
parameters of modules, each a module’s path under params as a parameter’s name begins
(parameter_name), ahead of its own patterns: the groups a run records and trains in, its
modules drawn fresh beside a loaded base (Objective.fresh_modules). Refused where such a group
then matches nothing, in a run that draws nothing fresh.
OptimConfig.rates
Section titled “OptimConfig.rates”def rates() -> dict[str | None, dict[str, float]]The rate fields each of build’s solvers holds in its state
(dew.optim.scheduled_solver), by param group, or under None
for the one solver of a config without groups. A resumed run sets
them over its checkpoint’s (with_rates), so the configured
rate is the one it steps at.
ParamGroup
Section titled “ParamGroup”class ParamGroup( name: str, patterns: tuple[str, ...] = (), learning_rate_multiplier: float = 1.0, weight_decay: float | None = None, spared: bool = False, schedule: ScheduleBase | None = None, b1: ScheduleBase | None = None, bounds: tuple[float, float] | None = None, fresh: bool = False,)A group of parameters the optimizer updates with their own learning rate, momentum, weight decay and bounds.
patterns are fnmatch patterns matched against a parameter’s path,
which is its dict keys joined by ’/’ (layers_3/self_attn/q_proj/kernel);
* also matches ’/’. A parameter belongs to the first group in
OptimConfig.param_groups that has a pattern it matches, as in
lm-engine, and a parameter that matches no group raises ValueError.
The group’s learning rate is its own schedule (the config’s learning_rate when
None) times learning_rate_multiplier. b1 schedules the first-moment
decay of adam, adamw and lamb, as torch’s OneCycleLR(cycle_momentum=True)
cycles Adam’s beta1; None keeps the optimizer’s own. weight_decay
replaces the config’s weight decay; None keeps it. For ‘adam’ that decay
is torch’s coupled L2 (_coupled_decay), and for ‘adamw’ the decoupled
one. bounds clamps each parameter into [lower, upper] after every
update (_keep_within).
spared: bool-
Whether the group’s decay spares its biases and vectors (
weight_decay_mask). A group decays every parameter it claims unless it does, as a torch optimizer’s parameter group does, so a run’s groups mirror the reference’s own groups (lm_engine’s, a torch recipe’s); the split an ungrouped optimizer takes (OptimConfig.spared) is the groups’ only where they say so. fresh: bool-
Whether the group claims the modules its run draws fresh beside a loaded base, ahead of its own
patterns(Objective.fresh_modules): whatOptimConfig.with_freshnames before a run records the group.
ParamGroup.rates
Section titled “ParamGroup.rates”def rates(learning_rate: float | ScheduleBase) -> dict[str, float]This group’s rate-valued fields (RateState): its own schedule’s, else the config’s
learning_rate’s, times learning_rate_multiplier.
ParamGroup.rate
Section titled “ParamGroup.rate”def rate(learning_rate: R) -> R | ScheduleBaseThe rate its solver runs at: its own schedule, or the config’s learning_rate where it
names none.
ParamGroup.solver
Section titled “ParamGroup.solver”def solver( make: Callable[..., optax.GradientTransformation], learning_rate: float | ScheduleBase, steps: int | None, opts: Mapping[str, object], *, clocked: bool = False,) -> optax.GradientTransformationBuild this group’s optimizer: make at the group’s rate, its fields held in state
(scheduled_solver), with the group’s b1 schedule and bounds, over a run of steps
updates, or of the progress of a clocked one.
A b1 schedule replaces any b1 in opts, and needs a make that
names a b1 argument (adam, adamw, lamb; not muon).
ParamGroup.mup
Section titled “ParamGroup.mup”def mup(width_multiplier: float) -> tuple[ParamGroup, ...]Return lm-engine’s muP parameter groups, in lm-engine’s order.
The groups are: norms, biases and dt_bias at the base rate without
weight decay; the token embeddings at the base rate with weight decay;
and everything else, including the router, A_log, D and the conv
taps, at the base rate divided by width_multiplier.
width_multiplier is lm-engine’s m_width, which is the model’s
logits_scaling.
class Power( peak_value: float, warmup_steps: int | str, a: float, b: float = -0.51, c: float = 1.0, tail: PowerTail | None = None, *, every: int = 1,)lm-engine’s power schedule (power_schedule), a power law of the step after a linear warmup.
After the warmup, the rate is min(peak_value, a * (step * c) ** b). With a
tail, a linear decay follows the power law, as in the last 29% of
Rigel’s run. lm-engine’s examples set a to 4 * batch size and c to
the tokens per step.
warmup_steps: int | str-
Its warmup in its own steps, or a share of the run such as
5%(ScheduleBase._warmup).
Power.warmup
Section titled “Power.warmup”def warmup(steps: int | None) -> intPower.peak
Section titled “Power.peak”def peak() -> floatpeak_value, the cap the power law climbs to.
Power.run_relative
Section titled “Power.run_relative”def run_relative() -> boolPower.written_warmup
Section titled “Power.written_warmup”def written_warmup() -> int | str | float | NonePower.marks
Section titled “Power.marks”def marks() -> dict[str, int]Power.progressed
Section titled “Power.progressed”def progressed( rates: Rates, progress: jax.Array, updates: jax.Array, marks: Mapping[str, jax.Array],) -> jax.ArrayIts power law of its updates after its warmup, and its tail from tail.start to the run’s end.
A warmup written as a share of the run rises over that share of the progress to the law’s value
at the update it reaches, the law scaled by how far through it the run stands.
Power.values
Section titled “Power.values”def values(steps: int | None, rates: Rates | None = None) -> optax.SchedulePowerProfilesState
Section titled “PowerProfilesState”class PowerProfilesState(NamedTuple)The optimizer state of power_profiles, kept beside the wrapped solver’s state.
updates: jax.Array-
The number of updates made so far, the t in the averages’ power-function profiles.
stds: jax.Array-
The relative standard deviation of each average.
powers: jax.Array-
Each average’s γ + 1, the power its decay raises (1 - 1/t) to, solved from its std when the state starts: a resumed average decays as it was tracked, whatever stds the resuming run configures.
PowerTail
Section titled “PowerTail”class PowerTail(start: int, steps: int | None = None, end_value: float = 0.0)The linear tail of a power schedule, from the power law’s rate at step
start to end_value at step steps, or at the run’s end when that is None.
QKClipState
Section titled “QKClipState”class QKClipState(NamedTuple)The rescale QK-Clip leaves for the stepped weights (stepped): the updates the clip has taken
part in, steps, and each query and key leaf’s factor a column of its last axis by its path,
every factor one before the first update or where no maxima were sown, so the state keeps one
structure. A stage-local run’s leaf holds one entry a stored layer, by the layer’s path
(layers_of), as an unstacked tree holds it.
ScheduleBase
Section titled “ScheduleBase”class ScheduleBase(*, every: int = 1)The base of the schedule records, each of which holds its own fields and builds an optax schedule.
A run’s record names the schedule’s class and holds only the fields that
schedule reads; dew.registry.schedules holds the short aliases. A
subclass builds its values in values; schedule reads them. The fields
it marks metadata={'rate': True} are its rates (rates), which a solver
holds as operands (scheduled_solver) and values reads from rates, so
a run’s program does not depend on them; a subclass that marks none
implements values(steps) alone.
every: int-
The updates each of the schedule’s own steps lasts.
torch recipes step a scheduler once an epoch, and with
everyset to the updates of an epoch this one holds each value through an epoch as torch’s does. A run ofstepsupdates is thensteps // everyof the schedule’s own steps, and every field that counts steps (warmup_steps,decay_steps) counts those.
ScheduleBase.rates
Section titled “ScheduleBase.rates”def rates() -> dict[str, float]Each rate-valued field’s value (metadata={'rate': True}) by its dotted path, nested
records such as PowerTail’s too, a None field’s derived default resolved where the schedule
derives one (OneCycle’s) and left out otherwise.
ScheduleBase.schedule
Section titled “ScheduleBase.schedule”def schedule(steps: int | None, rates: Rates | None = None) -> optax.ScheduleReturn the value at each update of a run of steps updates, as an optax schedule, its
rate-valued fields rates’ (rates() by default): operands in a solver’s state, or the
configured values.
ScheduleBase.warmup
Section titled “ScheduleBase.warmup”def warmup(steps: int | None) -> int | NoneIts warmup in its own steps over a run of steps updates, its share of the run resolved
(_warmup), as a run’s plan records it; None for a schedule with no warmup of steps.
ScheduleBase.values
Section titled “ScheduleBase.values”def values(steps: int | None, rates: Rates | None = None) -> optax.ScheduleReturn the value at each of the schedule’s own steps, over steps of them, from its
rate-valued fields rates (rates()). A value derived from the rates alone, as a ratio of
two, is the configured fields’, which a solver’s operands keep: they differ from them by
one factor, a param group’s multiplier.
ScheduleBase.peak
Section titled “ScheduleBase.peak”def peak() -> floatThe highest value the schedule takes: its largest rate-valued field, a warmup’s or decay’s ends, where its formula reaches no higher.
ScheduleBase.run_relative
Section titled “ScheduleBase.run_relative”def run_relative() -> boolWhether its horizon is the run’s, so a run that counts no updates ahead (one bounded in
examples) reads it over the run’s progress (progressed); a schedule whose every horizon is its
own reads its updates there as anywhere.
ScheduleBase.written_warmup
Section titled “ScheduleBase.written_warmup”def written_warmup() -> int | str | float | NoneIts warmup as written, in its own steps or as a share of the run, which a run that counts no
updates ahead records as it is (OptimConfig.warmups); None where it has none.
ScheduleBase.marks
Section titled “ScheduleBase.marks”def marks() -> dict[str, int]The update counts its law on a run’s progress starts a piece at (progressed), by name: the
end of a warmup counted in updates, say. The run records where its progress stood as its updates
reached each (ProgressState.marks).
ScheduleBase.progressed
Section titled “ScheduleBase.progressed”def progressed( rates: Rates, progress: jax.Array, updates: jax.Array, marks: Mapping[str, jax.Array],) -> jax.ArrayIts value where a run that counts no updates ahead stands: at progress, the share of the
run’s bound it has consumed (dew.state.TrainState.consumed), after updates updates, the
progress it stood at as it reached each count it marks. Each piece written in updates (a
warmup of 1000) reads the updates, and each that lasts to the run’s end reads the progress from
where it starts, its law kept: a share of the run (5%) is that share of the progress, and the
run’s end its whole. A schedule’s own steps (every) step its updates alone.
ScheduleSpec
Section titled “ScheduleSpec”delta_gradient_descent
Section titled “delta_gradient_descent”def delta_gradient_descent( learning_rate: float | optax.Schedule,) -> optax.GradientTransformationExtraArgsDelta gradient descent (Behrouz et al., Nested Learning, arXiv 2512.24695, eqs. 56-57): each
matrix steps to the minimizer of |W x - u|**2 / 2 + |W - W_t|**2 / (2 eta), where x is the step’s
input and u = -dL/dy its output’s negative gradient, so the step forgets what W_t held along x.
By Sherman-Morrison the minimizer is W_t (I - eta' x x^T) - eta' dL/dW with
eta' = eta / (1 + eta |x|**2), the paper’s eta / (1 + eta) at its normalized |x| = 1. A kernel
holds its input on axis 0, as a Linen kernel does (y = x K), so its step is
-eta' (x (x^T K) + dL/dK) over its other axes. The update takes inputs, a tree of x per leaf,
the parameters’ structure with None where a leaf has none: that leaf steps by gradient descent, the
rule at x = 0. An update given no inputs, or no parameters, is refused rather than run as plain
gradient descent.
scale_by_m3
Section titled “scale_by_m3”def scale_by_m3( *, beta1: float, beta2: float, beta3: float, alpha: float, eps: float, frequency: int, ns_steps: int = 5, weight_dimension_numbers=None, ns_precision: jax.lax.DotAlgorithmPreset | str | None = None,) -> optax.GradientTransformationMulti-scale Momentum Muon, M3 (Behrouz et al., Nested Learning, arXiv 2512.24695, Algorithm 1):
the direction (O1 + alpha O2) / sqrt(V + eps), which optax.scale_by_learning_rate steps by.
Each update adds beta1 g to the fast momentum M1 and beta2 g**2 to the second moment V, the
algorithm’s sums, neither decaying, and orthogonalizes M1 by ns_steps Newton-Schulz iterations,
Muon’s in the run’s numerics or at ns_precision’s dot algorithm, each matrix whole on one device and
the matrices spread over the mesh
(orthogonalized), each leaf shaped by its weight_dimension_numbers, a tree or a function of the
updates as Muon’s are; a leaf with none is one matrix, its rows the reduction. Its state is
kept in float32 at least (at_least_fp32), so a bfloat16 matrix’s momenta and gradients sum without
rounding to its width, and the directions come back in that dtype. Once every
frequency updates the slow momentum M2 adds beta3 times the chunk’s gradients and is
orthogonalized, O2, which the next chunk’s updates read: the first chunk’s is zero. The algorithm’s
sum, over i = (k-1) f .. k f, and eq. 75’s, over i = t - C .. t, each hold f + 1 gradients, the one
at a chunk’s boundary in two chunks; here each enters M2 once, with its own chunk’s. The paper states
no values for its factors, so each is the caller’s.
scale_by_trust_ratio
Section titled “scale_by_trust_ratio”def scale_by_trust_ratio( min_norm: float = 0.0, trust_coefficient: float = 1.0, eps: float = 0.0,) -> optax.GradientTransformationoptax’s scale_by_trust_ratio, LAMB’s (You et al., arXiv 1904.00962), each stored layer’s
own: a stage-local run’s leaf takes one ratio a layer, of that layer’s norms (layers_of), and
any other leaf its own, as optax’s does.
stepped
Section titled “stepped”def stepped(params, updates, before, after)params moved by updates, then each post-step effect of the update that took the optimizer’s
state from before to after, in the chain’s order: QK-Clip’s rescale (QKClipState), one
rounding of gamma * (W + update), and a bounded group’s clamp (BoundsState), exact. The
weights a step stores, and the only way to form them (the trainer’s transaction,
power_profiles’ averages). An effect whose transform the update did not run, a group a mask
left out or an update a wrapper deferred or rejected (optax.MultiSteps, optax.apply_if_finite),
whose state comes back as it was, leaves the weights as the update moved them.