dew.objectives.perturbation
Training without backpropagation: node perturbation, as Dust trains a transformer.
Dust (Q Labs, qlabs.sh/research/dust) trains with forward passes alone. Each linear layer’s output y_t is jittered, y_t + sigma a_t with a_t ~ N(0, I), independently at every unit of the batch (a token, an example), so one forward is a population of jitters; the change it makes in each unit’s own loss is that jitter’s reward, and the reward-weighted noise averaged over draws estimates the loss’s derivative at the layer’s output,
g_t = 1 / (2 K sigma) sum_draws (l_t(y + sigma a) - l_t(y - sigma a)) a_t,here from antithetic pairs, which cancel the reward’s baseline. A layer’s weight gradient is then the
product of that estimate with the layer’s input from the clean forward, summed over units, as
backpropagation forms it, and an embedding’s is the estimate scattered into the rows its ids read. Every
other parameter (a norm’s scale, a mixing scalar, an expert’s weights) is estimated by two-sided
weight-space evolution strategies, rewarded by the whole objective, and so is each use of a site’s
parameter that no layer call makes: a vocabulary head the loss reads as a matrix and contracts itself
(AffineHead.output_table), a tied embedding’s, whose matrix is jittered in that read alone and whose
estimate is added to the lookup’s, so a vocabulary row the batch only predicts is estimated too. A
wrapper’s read that delegates to its decoder’s is one read, its outermost, jittered and counted once.
NodePerturbation, mixed into any objective over a Flax model,
class ForwardLM(NodePerturbation, LMObjective): ...ForwardLM(model, seq_len, draws=256, sigma=0.1)does this for the objective’s own loss: its value is the objective’s, its statistics tree the objective’s
(a Ratio, a routed decoder’s RoutedStatistics), and only its derivative in the trained parameters is
estimated. The estimate is taken where the trainer asks for that derivative, under the cotangent the
trainer’s reduction gives the statistics, pooled over the update’s microbatches where it pools them, so
each statistic is weighted as the reduction weighs it and every denominator stays out. Each unit of the
first statistic is credited by its own share (Objective.scored_units: a supervised model’s examples, a
language model’s tokens with their configured terms) where a site’s output’s leading axes are the units’,
and every other statistic’s change, a router’s balance term, is credited to every unit. Units that read
other units’ outputs, a causal model’s tokens, are credited by their own share alone, as Dust’s direct
draws are: a lower-variance estimate, biased by what a jitter changes in later tokens.
Every pass reads the step’s own draws (dropout, routing noise), and the model’s other collections are
left as the clean loss leaves them. Each jitter is drawn in float32 and cast to its site’s output dtype,
and the cast draw is both the jitter and what its reward multiplies. A site whose output’s leading axis is
the batch’s rows draws each example’s jitter by its place among the update’s examples (Step.sample),
and every other draw, a weight’s or an unbatched site’s, by the update’s key, so a split or accumulated
update estimates what its batch whole does. A model that chooses its remat (RematChoosing) is estimated
without it, which a forward-only pass never needs; one that maps or scans its layers (nn.scan,
jax.vmap in its call) hides their calls’ values from a capture outside them, and is refused.
Dust’s own tuning, a draw budget per layer group, the attention internals scored through the attention output’s estimate, the head jittered on cached logits a vocabulary slab at a time, are the paper’s efficiency for its GPT and are not here: every site is jittered in every draw.
| Name | Summary |
|---|---|
NOISE | The dtype every jitter is drawn in, whatever its site’s: a float64 twin of a run draws what the run draws, and a bfloat16 site jitters by the draw cast to bfloat16. |
Path | |
Site | One Dense, DenseGeneral or Embed call of the model in the clean forward: its module’s path, how it maps (kind: “dense”, “embed” or “attend”), its input, its output’s shape and dtype, and a dense call’s contracted input axes, normalized and sorted, as its kernel’s leading axes take them. |
TableUse | One read of a model’s vocabulary head as a matrix (AffineHead.output_table), which a loss contracts itself rather than calling a layer: the unbound module that read it and its path in the model, so the read can be taken again over other parameters, and the matrix’s shape and dtype. |
NodePerturbation | Trains the objective it is mixed into without backpropagation (the module’s docstring), by draws antithetic pairs of output jitters of scale sigma a step, and weight_draws pairs of weight jitters of scale weight_sigma for the parameters no site holds. |
NOISE = jnp.float32The dtype every jitter is drawn in, whatever its site’s: a float64 twin of a run draws what the run draws, and a bfloat16 site jitters by the draw cast to bfloat16.
Path = tuple[str, ...]class Site( path: Path, kind: str, inputs: jax.Array, shape: tuple[int, ...], dtype: jnp.dtype, contracted: tuple[int, ...] = (),)One Dense, DenseGeneral or Embed call of the model in the clean forward: its module’s path,
how it maps (kind: “dense”, “embed” or “attend”), its input, its output’s shape and dtype, and a
dense call’s contracted input axes, normalized and sorted, as its kernel’s leading axes take them.
TableUse
Section titled “TableUse”class TableUse(module: nn.Module, path: Path, shape: tuple[int, ...], dtype: jnp.dtype)One read of a model’s vocabulary head as a matrix (AffineHead.output_table), which a loss
contracts itself rather than calling a layer: the unbound module that read it and its path in the
model, so the read can be taken again over other parameters, and the matrix’s shape and dtype.
NodePerturbation
Section titled “NodePerturbation”class NodePerturbation( *args=(), draws: int = 64, sigma: float = 0.2, weight_draws: int = 8, weight_sigma: float = 0.03, **kwargs={},)Trains the objective it is mixed into without backpropagation (the module’s docstring), by draws
antithetic pairs of output jitters of scale sigma a step, and weight_draws pairs of weight jitters
of scale weight_sigma for the parameters no site holds. Everything else, the model, its data and its
saved task, is the objective’s own, whose constructor takes the other arguments.
NodePerturbation.loss
Section titled “NodePerturbation.loss”def loss(variables: Variables, batch: Batch, step: Step)NodePerturbation.estimate
Section titled “NodePerturbation.estimate”def estimate( params: Variables, held: Variables, batch: Batch, step: Step, cotangent,) -> VariablesThe estimated derivative in every trained parameter of the statistics’ inner product with
cotangent, in the parameters’ tree: what the trainer’s reduction asks for.