Skip to content

dew.objectives.routing

How a run trains its routers: the one policy every objective composes.

A mixture’s routers choose each token’s experts (dew.nn.moe.Router). A run trains them by up to three terms, each optional, which ExpertBalance holds: the aux-loss-free balancing bias (balance_rate, arXiv 2408.15664), DeepSeek V2’s expert-level balance loss (aux_loss_alpha and seq_aux), and ST-MoE’s router z-loss (router_z_loss, arXiv 2202.08906).

An objective composes the policy with its own loss. Its model call opens the routers’ collection (dew.nn.moe.ROUTER_COLLECTION), compose adds the balance and z-loss to its statistics (RoutedStatistics) and counts the slots each router selected for the bias, and the commit applies those counts once per update (apply). The layer’s math stays in dew.nn.moe: the balance loss’s moments and the bias update’s arithmetic. The counting is here, once.

NameSummary
ExpertBalanceHow a run trains its routers, each term optional.
RoutedStatisticsA prediction’s statistics next to the routers’ own terms.
apply_routedvariables with the router counts balance took applied (ExpertBalance.apply).
reduce_routedThe objective value of a loss’s statistics, and whether they have any mass.
refuse_fixed_routerRefuse a run that would train routers no term of its loss reaches: a model’s routers whose expert weights are fixed (ExpertRouted.fixed_routers, a softmax top-1 router renormalized over its one expert), with no balance loss or z-loss to train them, one among the parameters the run trains (a parameter of its in held variables’ params; a tree drawn later trains them all).
router_countsCount the slots each router the forward ran selected, at the path of the balancing bias it keeps in moe.
router_momentsReturn one router’s routed-position statistics over batch’s rows (Objective.row_mean).
router_scoresCollect every router’s sown (scores, indices), in the tree’s key order.
router_z_termsReturn ST-MoE’s router z-loss, one Ratio per router.

dataclass source

class ExpertBalance(
balance_rate: float | None = None,
aux_loss_alpha: float | None = None,
seq_aux: bool = True,
router_z_loss: float = 0.0,
)

How a run trains its routers, each term optional.

balance_rate moves each router’s balancing bias against its load by this much at every committed update, which is DeepSeek’s aux-loss-free balancing. The model keeps the bias when its mixture sets bias=True. None leaves the bias where it is.

aux_loss_alpha scales DeepSeek V2’s expert-level balance loss (dew.nn.moe.global_router_loss), summed over the routers. seq_aux chooses its per-sequence form (dew.nn.moe.sequence_router_moments): a packed row’s sequences are its documents, each balanced alone, and their losses are averaged, as DeepSeek defines it. Megatron-LM and MaxText balance a packed row as one sequence, so their packed loss differs. None adds no balance loss.

router_z_loss is the routers’ own z-loss (ST-MoE): this coefficient times the squared log partition of every router’s gate logits, averaged over the positions each router saw in the step and summed over the routers. lm-engine’s MoE adds 0.1 of it to its switch loss before router_aux_loss_coef scales the sum, which matches router_z_loss = 0.1 * aux_loss_alpha here. Zero adds nothing.

def terms(
prediction: Ratio,
routing: Variables,
batch: Batch,
queries: jax.Array,
fields: Mapping[str, jax.Array | Coordinates],
) -> tuple[Ratio | RoutedStatistics, dict[str, jax.Array]]

The statistics the balance loss and the z-loss add to prediction, and the metrics they report.

routing is what the routers sowed, queries marks the router positions that count, and fields are the token fields of the batch’s rows, which name each row’s documents and the packed width of a coordinates column.

def compose(
params: Variables,
prediction: Ratio,
routing: Variables,
batch: Batch,
queries: jax.Array,
fields: Mapping[str, jax.Array | Coordinates],
) -> tuple[Ratio | RoutedStatistics, Variables | None, dict[str, jax.Array]]

The statistics an objective’s prediction takes from this policy, the router counts its commit applies (None where balance_rate is unset), and the metrics both report.

params holds the biases the counts are read against (effects); routing, queries and fields are the forward’s, as terms and effects take them.

def effects(
params: Variables,
routing: Variables,
queries: jax.Array,
fields: Mapping[str, jax.Array | Coordinates],
) -> tuple[Variables | None, dict[str, jax.Array]]

The bias counts of the routers the forward ran, and the spread of their load.

The counts are the effect the commit applies once per update (apply). The report is each router’s busiest and idlest share of the batch. Both are None where balance_rate is unset.

def reduce(stats: RoutedStatistics) -> tuple[jax.Array, jax.Array]

The objective value of routed statistics, and whether they have any mass.

The value is the prediction’s plus every router term’s.

def apply(variables: Variables, effects: Variables) -> Variables

variables with each counted router’s bias moved by its counts (dew.nn.moe.load_balance_update).

Only the moe collection is returned, which is the one a balancing step rewrites.

dataclass source

class RoutedStatistics(router_z: tuple[Ratio, ...] = ())

A prediction’s statistics next to the routers’ own terms.

Each router term normalises by its own count, not by the counted tokens, so the terms travel as separate statistics until ExpertBalance.reduce adds them up.

function source

def apply_routed(
balance: ExpertBalance | None,
variables: Variables,
effects: Variables,
) -> Variables

variables with the router counts balance took applied (ExpertBalance.apply).

function source

def reduce_routed(
balance: ExpertBalance | None,
stats: Ratio | RoutedStatistics,
) -> tuple[jax.Array, jax.Array]

The objective value of a loss’s statistics, and whether they have any mass.

A Ratio reduces by its own mean, and routed statistics by balance (ExpertBalance.reduce), which a routed loss has to name.

function source

def refuse_fixed_router(
model: object,
balance: ExpertBalance | None,
variables: Variables | None,
) -> None

Refuse a run that would train routers no term of its loss reaches: a model’s routers whose expert weights are fixed (ExpertRouted.fixed_routers, a softmax top-1 router renormalized over its one expert), with no balance loss or z-loss to train them, one among the parameters the run trains (a parameter of its in held variables’ params; a tree drawn later trains them all). Its routers would keep their draw, silently; the chosen expert’s softmax score weighs its output where norm_topk_prob=False, as Switch Transformer’s router does, and a router learns through it.

function source

def router_counts(
moe: Variables,
routing: Variables,
queries: jax.Array | None = None,
documents: jax.Array | None = None,
) -> Variables

Count the slots each router the forward ran selected, at the path of the balancing bias it keeps in moe.

The routers that ran are the ones routing holds sown indices for, so a router the forward skipped (a prediction depth left out, a draft stage) counts nothing; one that selects by a hash table keeps no bias and counts nothing either.

function source

def router_moments(
scores: jax.Array,
indices: jax.Array,
batch: Batch,
queries: jax.Array | None = None,
documents: jax.Array | None = None,
) -> RouterMoments

Return one router’s routed-position statistics over batch’s rows (Objective.row_mean).

scores is [rows, positions, experts] and indices the experts each position chose, [rows, positions, top_k].

function source

def router_scores(routing: Variables) -> list[tuple[jax.Array, jax.Array]]

Collect every router’s sown (scores, indices), in the tree’s key order.

function source

def router_z_terms(
routing: Variables,
weight: float,
batch: Batch,
queries: jax.Array | None = None,
documents: jax.Array | None = None,
) -> tuple[Ratio, ...]

Return ST-MoE’s router z-loss, one Ratio per router.

This is eq. 5 of ST-MoE (arXiv 2202.08906): weight times the squared log partition of the gate logits, summed over the positions the router saw and divided by their count. The count adds across micro-batches, so a step’s term is the mean over its routed positions, and the routers’ terms add up, as lm-engine’s per-layer (logsumexp(logits) ** 2).mean() does. The positions are batch’s rows’ (Objective.row_mean).