Skip to content

dew.nn.mlp

The feed-forward every family builds: GatedMLP, two maps with an activation between or three with a gate, which owns its maps’ names, placement, remat names, activation rounding and work count.

NameSummary
GatedMLPComputes the gated feed-forward down_proj(act(gate_proj(x)) * up_proj(x)).
geglu_feed_forwarddiffusers’ FeedForward: a GEGLU to four times dim and back, biased, its gate and up projections packed as the checkpoint keeps them, the up projection first; approximate is the gate’s tanh form of GELU, else the erf form, and fields any other GatedMLP field.

Flax module source

class GatedMLP(
activation: GatedActivation | Activation = 'swiglu',
use_bias: bool = False,
activation_sparsity: float = 0.0,
swiglu_limit: float | None = None,
init_std: float | None = None,
output_init_std: float | None = None,
packed: bool = False,
packed_order: tuple[str, str] = ('gate_proj', 'up_proj'),
linear: Callable[..., nn.Module] = functools.partial(Dense, dot_general=float32_dot_general),
dropout_rate: float = 0.0,
ngpt: NGPT | None = None,
dtype: Dtype | None = None,
precision: PrecisionLike = None,
in_features: int | None = None,
activation_after: bool = False,
names: tuple[str, str] = ('up_proj', 'down_proj'),
named_axes: tuple[tuple[str, LogicalAxes], ...] = (),
gate_multiplier: float = 1.0,
down_multiplier: float = 1.0,
)

Computes the gated feed-forward down_proj(act(gate_proj(x)) * up_proj(x)).

activation picks act: swiglu is silu, geglu is the tanh approximation of gelu (HF’s gelu_pytorch_tanh), and geglu_exact is the erf form (HF’s gelu). A Situ is Kimi K3’s SiTU, which transforms both halves (dew.nn.activations.gated_product). The UNGATED activations, by name, or an activation function (dew.nn.activations) build the ungated MLP with two projections.

activation_sparsity is Gemma 3n’s gaussian top-k on the gate (gaussian_topk). swiglu_limit is the clamp that GLM-5.3-Flash and DeepSeek V4 apply before the activation: the gate is capped from above, and the up projection on both sides. Both need a gated activation. packed keeps the gate and up projections in one gate_up_proj from init on, as DeepSeek-V4.1’s vision tower stores them, in packed_order (diffusers’ GEGLU stores the up projection first), and linear is the projections’ module (Gemma 4’s vision tower clips theirs). Where the generation’s kernel row measured it faster (‘gate_up’: ‘joined’), separate gate and up projections run as one GEMM over their kernels joined per call (dew.nn.joined). The product reads a packed or joined output in place (packed_gated_product). dropout_rate drops the activated hidden units before the down projection while training, as T5’s T5DenseActDense does.

The ungated MLP may read another width than it writes (in_features, which the count reads), apply its activation after the down projection too (activation_after, a diffusion time embedding’s or a vision adapter’s), and store its two projections under the names its checkpoint uses (names, up then down).

use_bias: bool

Whether its projections add a bias: none by default, as Llama’s and most published gated feed-forwards (mlp_bias: false), where Flax’s Dense adds one.

linear: Callable[..., nn.Module]

How each projection is built: a factory of dew.nn.dense.Dense’s signature, which declares each projection’s axes where it draws it (kernel_axes).

ngpt: NGPT | None

The normalized transformer’s gains (dew.nn.ngpt.NGPT): the gate and the up projection each take a learned per-feature gain (suv_gate, suv_up) and the gated product is unit-normed before the down projection; None for none.

named_axes: tuple[tuple[str, LogicalAxes], ...]

The logical axes of projections under names of the owner’s own (names), beside MLP_AXES.

gate_multiplier: float

A gated MLP’s gate projection’s output times this before the activation, and down_multiplier its down projection’s output: Falcon-H1’s mlp_multipliers, each a product rounded once to its operand’s dtype (dew.nn.precision.scaled), as torch’s product of a tensor and a Python float.

reads_train: bool

Whether its call takes train (ReadsTrain): it drops hidden units while training.

def model_flops(lengths) -> float

The residual-width MLP’s two maps, or three with a learned gate.

def projections() -> dict[str, tuple[int, int]]

The feed-forward’s maps as {name: (inputs, outputs)}: what setup builds, the count reads and the packed group (_group) concatenates.

def setup()
def projection_groups() -> tuple[ProjectionGroup, ...]

Its group (_group) where it is gated and its variables hold it (ProjectionSites), which setup reads packed in the members’ place.

function source

def geglu_feed_forward(dim: int, *, approximate: bool = True, **fields={}) -> GatedMLP

diffusers’ FeedForward: a GEGLU to four times dim and back, biased, its gate and up projections packed as the checkpoint keeps them, the up projection first; approximate is the gate’s tanh form of GELU, else the erf form, and fields any other GatedMLP field.