dew.nn.gemma3n
Gemma 3n’s residual stream: AltUp, the LAuReL block and activation sparsity.
Gemma3nTextModel (modeling_gemma3n.py) keeps altup_num_inputs copies of
the residual. The first copy is the embeddings, and each other copy is its
own projection of them, rescaled to the embeddings’ RMS. Every layer
predicts all copies from the active one (Gemma3nTextAltUp.predict), runs
the block on that prediction, corrects every copy by the block’s innovation
(correct), and adds the per-layer input to the copies past the first.
After the last layer, the copies are projected back and rescaled to the
first copy’s magnitude, and their mean goes into the final norm.
Gemma3nTextLaurelBlock is a rank-laurel_rank map of the block’s normed
input, normed and added back; its sum with the attention residual is divided
by sqrt(2). Gemma3nTextMLP._gaussian_topk keeps the gate activations that
are more than norm.ppf(sparsity) standard deviations above the mean, which any
gated MLP runs (dew.nn.mlp.gaussian_topk).
| Name | Summary |
|---|---|
MAGNITUDE_EPSILON | |
AltUp | How many copies of the residual stream a model keeps, and which one its blocks run on. |
AltUpLayer | Runs one layer’s AltUp: predict before the block and correct after it. |
LaurelBlock | Computes Gemma 3n’s learned augmented residual (LAuReL). |
clamped_predictions | params with every AltUp layer’s stored prediction coefficients clamped to +-clip, as the reference clamps prediction_coefs.weight in place before a training forward (modeling_gemma3n.py:1027-1028), so the optimizer updates the clamped weight. |
rescale_to | Return x scaled per token to target’s RMS magnitude. |
MAGNITUDE_EPSILON
Section titled “MAGNITUDE_EPSILON”MAGNITUDE_EPSILON = 1e-05class AltUp( num_inputs: int = 4, active_idx: int = 0, coef_clip: float | None = 120.0, correct_scale: bool = True,)How many copies of the residual stream a model keeps, and which one its blocks run on.
The fields are the reference’s altup_* config entries without the
prefix (configuration_gemma3n.py).
AltUpLayer
Section titled “AltUpLayer”class AltUpLayer( norm_eps: float | None = 1e-06, dtype: Dtype | None = None, precision: PrecisionLike = None,)Runs one layer’s AltUp: predict before the block and correct after it.
The stream is [num_inputs, B, S, D]. Both steps read a token’s
modalities: the tanh of the modality_router output for the token’s
normed, 1 / D-scaled active copy. predict maps the modalities to a
num_inputs by num_inputs matrix per token (prediction_coefs) that
mixes the copies, and adds the mix to the copies. correct maps the
modalities to one coefficient per copy (correction_coefs, plus one).
Each coefficient scales the block’s innovation, which is the activated
output minus the active prediction, and the scaled innovation is added to
every prediction. coef_clip bounds both coefficient weights during
training, as the reference clamps them (_Coefficients).
AltUpLayer.projections
Section titled “AltUpLayer.projections”def projections() -> dict[str, tuple[int, int]]AltUpLayer.model_flops
Section titled “AltUpLayer.model_flops”def model_flops(lengths) -> floatBoth router reads, coefficient maps and the prediction’s copy contraction.
AltUpLayer.projection_uses
Section titled “AltUpLayer.projection_uses”def projection_uses() -> dict[str, int]AltUpLayer.parameter_flops
Section titled “AltUpLayer.parameter_flops”def parameter_flops(lengths) -> floatAltUpLayer.mixing_flops
Section titled “AltUpLayer.mixing_flops”def mixing_flops(lengths) -> floatAltUpLayer.setup
Section titled “AltUpLayer.setup”def setup()AltUpLayer.modalities
Section titled “AltUpLayer.modalities”def modalities(x)AltUpLayer.predict
Section titled “AltUpLayer.predict”def predict(stream, train: bool = False)AltUpLayer.correct
Section titled “AltUpLayer.correct”def correct(predictions, activated, train: bool = False)AltUpLayer.scale_corrected_output
Section titled “AltUpLayer.scale_corrected_output”def scale_corrected_output(corrected)LaurelBlock
Section titled “LaurelBlock”class LaurelBlock( norm_eps: float | None = 1e-06, dtype: Dtype | None = None, precision: PrecisionLike = None,)Computes Gemma 3n’s learned augmented residual (LAuReL).
The output is x + post_laurel_norm(linear_right(linear_left(x))).
LaurelBlock.model_flops
Section titled “LaurelBlock.model_flops”def model_flops(lengths) -> floatLaurelBlock.projections
Section titled “LaurelBlock.projections”def projections() -> dict[str, tuple[int, int]]clamped_predictions
Section titled “clamped_predictions”def clamped_predictions(params, clip: float)params with every AltUp layer’s stored prediction coefficients
clamped to +-clip, as the reference clamps prediction_coefs.weight in
place before a training forward (modeling_gemma3n.py:1027-1028), so the
optimizer updates the clamped weight. Any other leaf, and a tree holding
no AltUp layer, is returned as it is.
rescale_to
Section titled “rescale_to”def rescale_to(x, target)Return x scaled per token to target’s RMS magnitude.
The mean square of x is floored at MAGNITUDE_EPSILON before its square
root (the reference’s new_magnitude). A zero target (a zero pad
embedding at a padded slot) gets magnitude zero with a zero gradient (sqrt_or_zero).