Skip to content

dew.objectives.perturbation

Training without backpropagation: node perturbation, as Dust trains a transformer.

Dust (Q Labs, qlabs.sh/research/dust) trains with forward passes alone. Each linear layer’s output y_t is jittered, y_t + sigma a_t with a_t ~ N(0, I), independently at every unit of the batch (a token, an example), so one forward is a population of jitters; the change it makes in each unit’s own loss is that jitter’s reward, and the reward-weighted noise averaged over draws estimates the loss’s derivative at the layer’s output,

g_t = 1 / (2 K sigma) sum_draws (l_t(y + sigma a) - l_t(y - sigma a)) a_t,

here from antithetic pairs, which cancel the reward’s baseline. A layer’s weight gradient is then the product of that estimate with the layer’s input from the clean forward, summed over units, as backpropagation forms it, and an embedding’s is the estimate scattered into the rows its ids read. Every other parameter (a norm’s scale, a mixing scalar, an expert’s weights) is estimated by two-sided weight-space evolution strategies, rewarded by the whole objective, and so is each use of a site’s parameter that no layer call makes: a vocabulary head the loss reads as a matrix and contracts itself (AffineHead.output_table), a tied embedding’s, whose matrix is jittered in that read alone and whose estimate is added to the lookup’s, so a vocabulary row the batch only predicts is estimated too. A wrapper’s read that delegates to its decoder’s is one read, its outermost, jittered and counted once.

NodePerturbation, mixed into any objective over a Flax model,

class ForwardLM(NodePerturbation, LMObjective): ...
ForwardLM(model, seq_len, draws=256, sigma=0.1)

does this for the objective’s own loss: its value is the objective’s, its statistics tree the objective’s (a Ratio, a routed decoder’s RoutedStatistics), and only its derivative in the trained parameters is estimated. The estimate is taken where the trainer asks for that derivative, under the cotangent the trainer’s reduction gives the statistics, pooled over the update’s microbatches where it pools them, so each statistic is weighted as the reduction weighs it and every denominator stays out. Each unit of the first statistic is credited by its own share (Objective.scored_units: a supervised model’s examples, a language model’s tokens with their configured terms) where a site’s output’s leading axes are the units’, and every other statistic’s change, a router’s balance term, is credited to every unit. Units that read other units’ outputs, a causal model’s tokens, are credited by their own share alone, as Dust’s direct draws are: a lower-variance estimate, biased by what a jitter changes in later tokens.

Every pass reads the step’s own draws (dropout, routing noise), and the model’s other collections are left as the clean loss leaves them. Each jitter is drawn in float32 and cast to its site’s output dtype, and the cast draw is both the jitter and what its reward multiplies. A site whose output’s leading axis is the batch’s rows draws each example’s jitter by its place among the update’s examples (Step.sample), and every other draw, a weight’s or an unbatched site’s, by the update’s key, so a split or accumulated update estimates what its batch whole does. A model that chooses its remat (RematChoosing) is estimated without it, which a forward-only pass never needs; one that maps or scans its layers (nn.scan, jax.vmap in its call) hides their calls’ values from a capture outside them, and is refused.

Dust’s own tuning, a draw budget per layer group, the attention internals scored through the attention output’s estimate, the head jittered on cached logits a vocabulary slab at a time, are the paper’s efficiency for its GPT and are not here: every site is jittered in every draw.

NameSummary
NOISEThe dtype every jitter is drawn in, whatever its site’s: a float64 twin of a run draws what the run draws, and a bfloat16 site jitters by the draw cast to bfloat16.
Path
SiteOne Dense, DenseGeneral or Embed call of the model in the clean forward: its module’s path, how it maps (kind: “dense”, “embed” or “attend”), its input, its output’s shape and dtype, and a dense call’s contracted input axes, normalized and sorted, as its kernel’s leading axes take them.
TableUseOne read of a model’s vocabulary head as a matrix (AffineHead.output_table), which a loss contracts itself rather than calling a layer: the unbound module that read it and its path in the model, so the read can be taken again over other parameters, and the matrix’s shape and dtype.
NodePerturbationTrains the objective it is mixed into without backpropagation (the module’s docstring), by draws antithetic pairs of output jitters of scale sigma a step, and weight_draws pairs of weight jitters of scale weight_sigma for the parameters no site holds.

attribute source

NOISE = jnp.float32

The dtype every jitter is drawn in, whatever its site’s: a float64 twin of a run draws what the run draws, and a bfloat16 site jitters by the draw cast to bfloat16.

attribute source

Path = tuple[str, ...]

dataclass source

class Site(
path: Path,
kind: str,
inputs: jax.Array,
shape: tuple[int, ...],
dtype: jnp.dtype,
contracted: tuple[int, ...] = (),
)

One Dense, DenseGeneral or Embed call of the model in the clean forward: its module’s path, how it maps (kind: “dense”, “embed” or “attend”), its input, its output’s shape and dtype, and a dense call’s contracted input axes, normalized and sorted, as its kernel’s leading axes take them.

dataclass source

class TableUse(module: nn.Module, path: Path, shape: tuple[int, ...], dtype: jnp.dtype)

One read of a model’s vocabulary head as a matrix (AffineHead.output_table), which a loss contracts itself rather than calling a layer: the unbound module that read it and its path in the model, so the read can be taken again over other parameters, and the matrix’s shape and dtype.

class source

class NodePerturbation(
*args=(),
draws: int = 64,
sigma: float = 0.2,
weight_draws: int = 8,
weight_sigma: float = 0.03,
**kwargs={},
)

Trains the objective it is mixed into without backpropagation (the module’s docstring), by draws antithetic pairs of output jitters of scale sigma a step, and weight_draws pairs of weight jitters of scale weight_sigma for the parameters no site holds. Everything else, the model, its data and its saved task, is the objective’s own, whose constructor takes the other arguments.

def loss(variables: Variables, batch: Batch, step: Step)
def estimate(
params: Variables,
held: Variables,
batch: Batch,
step: Step,
cotangent,
) -> Variables

The estimated derivative in every trained parameter of the statistics’ inner product with cotangent, in the parameters’ tree: what the trainer’s reduction asks for.