Skip to content

dew.nn.mixers.parallel

The parallel mixer kind: several token mixers reading one normed input side by side, their outputs summed into the layer’s one residual add, as a parallel hybrid’s layer runs them (Falcon-H1’s Mamba-2 beside its attention, FalconH1DecoderLayer.forward in modeling_falcon_h1.py).

NameSummary
ParallelBranchOne mixer of a parallel layer: its module’s name under the layer’s mixer, the mixer kind it runs, and the factors its input and its output are scaled by (Falcon-H1’s attention_in_multiplier and ssm_in_multiplier on the inputs, attention_out_multiplier and ssm_out_multiplier on the outputs), each a product in at least float32 rounded once to its operand’s dtype (dew.nn.precision.scaled), as torch’s product of a tensor and a Python float.
ParallelMixersThe mixers of a parallel layer over one input: each reads the input times its branch’s input multiplier, under the layer’s call, and their outputs, each times its output multiplier, are summed in the branches’ order (ParallelMixer).
ParallelMixerSeveral mixer kinds side by side in one layer (ParallelMixers), each ParallelBranch its own mixer under its name, at the layer’s geometry: a parallel hybrid, Falcon-H1’s Mamba-2 beside its attention.

dataclass source

class ParallelBranch(
name: str,
mixer: MixerBase,
input_multiplier: float = 1.0,
output_multiplier: float = 1.0,
)

One mixer of a parallel layer: its module’s name under the layer’s mixer, the mixer kind it runs, and the factors its input and its output are scaled by (Falcon-H1’s attention_in_multiplier and ssm_in_multiplier on the inputs, attention_out_multiplier and ssm_out_multiplier on the outputs), each a product in at least float32 rounded once to its operand’s dtype (dew.nn.precision.scaled), as torch’s product of a tensor and a Python float.

Flax module source

class ParallelMixers()

The mixers of a parallel layer over one input: each reads the input times its branch’s input multiplier, under the layer’s call, and their outputs, each times its output multiplier, are summed in the branches’ order (ParallelMixer). Every branch holds its own parameters, its own decode cache and its own routers under its name. A prediction depth’s phase reaches the branches that read it (MixerBase.reads_phase), the rest running their ordinary call; a routing replay reaches the branches that route (RoutesExperts), each its own sites’ records by their paths below it (routed_sites, dew.nn.protocols.RoutesBySite), or the one routed site a layer’s single record.

branches: tuple[tuple[str, Callable[..., nn.Module], float, float, bool], ...]

Each branch’s name, its mixer’s factory, its input and output multipliers, and whether it reads a prediction phase.

reads: frozenset[str]

The MIXING fields every branch reads (dew.nn.backbones.decoder_block.ReadsMixing; one that declares none reads none): the call carries each to every branch, so a field one branch would not read is one the block refuses, rather than a restriction that branch would drop.

routes_experts: bool

Whether a branch routes over experts (dew.nn.protocols.RoutesExperts), so the layer’s routing replay reaches it.

routed_sites: frozenset[str]

Its routed sites’ paths below it: a routed branch’s name, or a composed branch’s sites below its name (dew.nn.protocols.RoutesBySite).

reads_train: bool

Whether a branch takes the train flag (dew.nn.protocols.ReadsTrain).

def setup()
def model_flops(lengths) -> float

Its branches’ work; the scalings and the sum are elementwise.

def parameter_flops(lengths) -> float

Its branches’ learned maps.

def projection_groups() -> tuple[ProjectionGroup, ...]

The packed groups its branches declare (dew.nn.protocols.ProjectionSites).

dataclass source

class ParallelMixer(branches: tuple[ParallelBranch, ...] = ())

Several mixer kinds side by side in one layer (ParallelMixers), each ParallelBranch its own mixer under its name, at the layer’s geometry: a parallel hybrid, Falcon-H1’s Mamba-2 beside its attention. What a branch keeps, reads or refuses, the layer keeps, reads or refuses: its cache, its layer’s index (reads_depth, which the call hands every branch), its remat marks, its mixed serving step, its generation, a prediction depth’s phase (reads_phase), and its routing replay, by its name.

def publishes(kv_shared: bool) -> bool
def marks() -> tuple[str, ...] | None

The names any branch marks; every name where one marks every name.

def mixed_refusal(ctx: MixerContext) -> str | None
def generation_refusal() -> str | None
def build(ctx: MixerContext) -> Callable[..., nn.Module]