Skip to content

dew.nn.ngpt

The normalized transformer’s (nGPT, arXiv 2410.01131) arithmetic: hidden states on the unit sphere, each sublayer’s output joining the residual by a learned step along the sphere.

NVIDIA’s Dory (vllm-project/vllm#60149, vllm/model_executor/models/dory.py at 55076950) is the reference these follow, rounding for rounding: a unit norm reduces in fp32 and returns its input’s dtype (justnorm), and the residual’s step norms both ends in that dtype before it takes the step in fp32 (slerp_norm). Each fp32 is at least fp32 (at_least_fp32): a float64 model computes in float64 throughout.

NameSummary
NGPTA normalized transformer’s constants, as a checkpoint states them: std the initializer’s standard deviation (initializer_range), alpha the residual’s initial step (ngpt_alpha_init) and gate_logit the attention output gate’s initial logit scale (attention_output_gate_logit_scale_init, None for the square root of the model’s width).

dataclass source

class NGPT(std: float, alpha: float, gate_logit: float | None = None)

A normalized transformer’s constants, as a checkpoint states them: std the initializer’s standard deviation (initializer_range), alpha the residual’s initial step (ngpt_alpha_init) and gate_logit the attention output gate’s initial logit scale (attention_output_gate_logit_scale_init, None for the square root of the model’s width).

nGPT stores each learned gain at its initializer’s scale and multiplies it back by a constant these give: the step’s alpha / std, the query’s sqrt(head_dim) / std, the gate’s sqrt(gate_logit) / std, the feed-forward’s sqrt(width) and the logits’ 1 / std. Each starts at that scale (start): std, or 1 for the feed-forward’s, so the step starts at alpha, the query’s squared gain at head_dim (nGPT’s sqrt(head_dim) logit scale over the usual 1 / sqrt(head_dim)), the gate’s at gate_logit, the feed-forward’s at sqrt(width) and the logits’ at 1, as nGPT’s init_scaling starts them; a zero alpha or gate_logit is a gain that starts and stays at zero.

A model under it keeps every hidden state on the unit sphere: no norm has a weight, the keys and the attention’s and feed-forward’s outputs are unit-normed (dew.nn.precision.l2norm), each sublayer joins the residual by stepped, and no norm precedes the head.

start: float

The stored value every gain but the feed-forward’s starts at.

def query(head_dim: int) -> float
def gate(width: int) -> float
def hidden(width: int) -> float