dew.objectives.routing
How a run trains its routers: the one policy every objective composes.
A mixture’s routers choose each token’s experts (dew.nn.moe.Router). A run
trains them by up to three terms, each optional, which ExpertBalance holds:
the aux-loss-free balancing bias (balance_rate, arXiv 2408.15664), DeepSeek
V2’s expert-level balance loss (aux_loss_alpha and seq_aux), and ST-MoE’s
router z-loss (router_z_loss, arXiv 2202.08906).
An objective composes the policy with its own loss. Its model call opens the
routers’ collection (dew.nn.moe.ROUTER_COLLECTION), compose adds the
balance and z-loss to its statistics (RoutedStatistics) and counts the slots
each router selected for the bias, and the commit applies those counts once
per update (apply). The layer’s math stays in dew.nn.moe: the balance
loss’s moments and the bias update’s arithmetic. The counting is here, once.
| Name | Summary |
|---|---|
ExpertBalance | How a run trains its routers, each term optional. |
RoutedStatistics | A prediction’s statistics next to the routers’ own terms. |
apply_routed | variables with the router counts balance took applied (ExpertBalance.apply). |
reduce_routed | The objective value of a loss’s statistics, and whether they have any mass. |
refuse_fixed_router | Refuse a run that would train routers no term of its loss reaches: a model’s routers whose expert weights are fixed (ExpertRouted.fixed_routers, a softmax top-1 router renormalized over its one expert), with no balance loss or z-loss to train them, one among the parameters the run trains (a parameter of its in held variables’ params; a tree drawn later trains them all). |
router_counts | Count the slots each router the forward ran selected, at the path of the balancing bias it keeps in moe. |
router_moments | Return one router’s routed-position statistics over batch’s rows (Objective.row_mean). |
router_scores | Collect every router’s sown (scores, indices), in the tree’s key order. |
router_z_terms | Return ST-MoE’s router z-loss, one Ratio per router. |
ExpertBalance
Section titled “ExpertBalance”class ExpertBalance( balance_rate: float | None = None, aux_loss_alpha: float | None = None, seq_aux: bool = True, router_z_loss: float = 0.0,)How a run trains its routers, each term optional.
balance_rate moves each router’s balancing bias against its load by this
much at every committed update, which is DeepSeek’s aux-loss-free
balancing. The model keeps the bias when its mixture sets bias=True.
None leaves the bias where it is.
aux_loss_alpha scales DeepSeek V2’s expert-level balance loss
(dew.nn.moe.global_router_loss), summed over the routers. seq_aux
chooses its per-sequence form (dew.nn.moe.sequence_router_moments): a
packed row’s sequences are its documents, each balanced alone, and their
losses are averaged, as DeepSeek defines it. Megatron-LM and MaxText
balance a packed row as one sequence, so their packed loss differs. None
adds no balance loss.
router_z_loss is the routers’ own z-loss (ST-MoE): this coefficient times
the squared log partition of every router’s gate logits, averaged over the
positions each router saw in the step and summed over the routers.
lm-engine’s MoE adds 0.1 of it to its switch loss before router_aux_loss_coef
scales the sum, which matches router_z_loss = 0.1 * aux_loss_alpha here. Zero
adds nothing.
ExpertBalance.terms
Section titled “ExpertBalance.terms”def terms( prediction: Ratio, routing: Variables, batch: Batch, queries: jax.Array, fields: Mapping[str, jax.Array | Coordinates],) -> tuple[Ratio | RoutedStatistics, dict[str, jax.Array]]The statistics the balance loss and the z-loss add to prediction, and the metrics they report.
routing is what the routers sowed, queries marks the router
positions that count, and fields are the token fields of the batch’s
rows, which name each row’s documents and the packed width of a
coordinates column.
ExpertBalance.compose
Section titled “ExpertBalance.compose”def compose( params: Variables, prediction: Ratio, routing: Variables, batch: Batch, queries: jax.Array, fields: Mapping[str, jax.Array | Coordinates],) -> tuple[Ratio | RoutedStatistics, Variables | None, dict[str, jax.Array]]The statistics an objective’s prediction takes from this policy, the router counts its commit
applies (None where balance_rate is unset), and the metrics both report.
params holds the biases the counts are read against (effects); routing, queries and fields
are the forward’s, as terms and effects take them.
ExpertBalance.effects
Section titled “ExpertBalance.effects”def effects( params: Variables, routing: Variables, queries: jax.Array, fields: Mapping[str, jax.Array | Coordinates],) -> tuple[Variables | None, dict[str, jax.Array]]The bias counts of the routers the forward ran, and the spread of their load.
The counts are the effect the commit applies once per update (apply).
The report is each router’s busiest and idlest share of the batch. Both
are None where balance_rate is unset.
ExpertBalance.reduce
Section titled “ExpertBalance.reduce”def reduce(stats: RoutedStatistics) -> tuple[jax.Array, jax.Array]The objective value of routed statistics, and whether they have any mass.
The value is the prediction’s plus every router term’s.
ExpertBalance.apply
Section titled “ExpertBalance.apply”def apply(variables: Variables, effects: Variables) -> Variablesvariables with each counted router’s bias moved by its counts (dew.nn.moe.load_balance_update).
Only the moe collection is returned, which is the one a balancing step rewrites.
RoutedStatistics
Section titled “RoutedStatistics”class RoutedStatistics(router_z: tuple[Ratio, ...] = ())A prediction’s statistics next to the routers’ own terms.
Each router term normalises by its own count, not by the counted tokens,
so the terms travel as separate statistics until ExpertBalance.reduce
adds them up.
apply_routed
Section titled “apply_routed”def apply_routed( balance: ExpertBalance | None, variables: Variables, effects: Variables,) -> Variablesvariables with the router counts balance took applied (ExpertBalance.apply).
reduce_routed
Section titled “reduce_routed”def reduce_routed( balance: ExpertBalance | None, stats: Ratio | RoutedStatistics,) -> tuple[jax.Array, jax.Array]The objective value of a loss’s statistics, and whether they have any mass.
A Ratio reduces by its own mean, and routed statistics by balance (ExpertBalance.reduce), which
a routed loss has to name.
refuse_fixed_router
Section titled “refuse_fixed_router”def refuse_fixed_router( model: object, balance: ExpertBalance | None, variables: Variables | None,) -> NoneRefuse a run that would train routers no term of its loss reaches: a model’s routers whose expert
weights are fixed (ExpertRouted.fixed_routers, a softmax top-1 router renormalized over its one expert),
with no balance loss or z-loss to train them, one among the parameters the run trains (a parameter
of its in held variables’ params; a tree drawn later trains them all). Its routers would keep
their draw, silently; the chosen expert’s softmax score weighs its output where norm_topk_prob=False,
as Switch Transformer’s router does, and a router learns through it.
router_counts
Section titled “router_counts”def router_counts( moe: Variables, routing: Variables, queries: jax.Array | None = None, documents: jax.Array | None = None,) -> VariablesCount the slots each router the forward ran selected, at the path of the balancing bias it keeps in
moe.
The routers that ran are the ones routing holds sown indices for, so
a router the forward skipped (a prediction depth left out, a draft stage)
counts nothing; one that selects by a hash table keeps no bias and counts
nothing either.
router_moments
Section titled “router_moments”def router_moments( scores: jax.Array, indices: jax.Array, batch: Batch, queries: jax.Array | None = None, documents: jax.Array | None = None,) -> RouterMomentsReturn one router’s routed-position statistics over batch’s rows (Objective.row_mean).
scores is [rows, positions, experts] and indices the experts each
position chose, [rows, positions, top_k].
router_scores
Section titled “router_scores”def router_scores(routing: Variables) -> list[tuple[jax.Array, jax.Array]]Collect every router’s sown (scores, indices), in the tree’s key order.
router_z_terms
Section titled “router_z_terms”def router_z_terms( routing: Variables, weight: float, batch: Batch, queries: jax.Array | None = None, documents: jax.Array | None = None,) -> tuple[Ratio, ...]Return ST-MoE’s router z-loss, one Ratio per router.
This is eq. 5 of ST-MoE (arXiv 2202.08906): weight times the squared log
partition of the gate logits, summed over the positions the router saw and
divided by their count. The count adds across micro-batches, so a step’s term
is the mean over its routed positions, and the routers’ terms add up, as
lm-engine’s per-layer (logsumexp(logits) ** 2).mean() does. The positions
are batch’s rows’ (Objective.row_mean).