dew.nn.mixers.parallel
The parallel mixer kind: several token mixers reading one normed input side by side, their outputs summed into the layer’s one residual add, as a parallel hybrid’s layer runs them (Falcon-H1’s Mamba-2 beside its attention, FalconH1DecoderLayer.forward in modeling_falcon_h1.py).
| Name | Summary |
|---|---|
ParallelBranch | One mixer of a parallel layer: its module’s name under the layer’s mixer, the mixer kind it runs, and the factors its input and its output are scaled by (Falcon-H1’s attention_in_multiplier and ssm_in_multiplier on the inputs, attention_out_multiplier and ssm_out_multiplier on the outputs), each a product in at least float32 rounded once to its operand’s dtype (dew.nn.precision.scaled), as torch’s product of a tensor and a Python float. |
ParallelMixers | The mixers of a parallel layer over one input: each reads the input times its branch’s input multiplier, under the layer’s call, and their outputs, each times its output multiplier, are summed in the branches’ order (ParallelMixer). |
ParallelMixer | Several mixer kinds side by side in one layer (ParallelMixers), each ParallelBranch its own mixer under its name, at the layer’s geometry: a parallel hybrid, Falcon-H1’s Mamba-2 beside its attention. |
ParallelBranch
Section titled “ParallelBranch”class ParallelBranch( name: str, mixer: MixerBase, input_multiplier: float = 1.0, output_multiplier: float = 1.0,)One mixer of a parallel layer: its module’s name under the layer’s mixer, the mixer kind it
runs, and the factors its input and its output are scaled by (Falcon-H1’s
attention_in_multiplier and ssm_in_multiplier on the inputs, attention_out_multiplier and
ssm_out_multiplier on the outputs), each a product in at least float32 rounded once to its
operand’s dtype (dew.nn.precision.scaled), as torch’s product of a tensor and a Python float.
ParallelMixers
Section titled “ParallelMixers”class ParallelMixers()The mixers of a parallel layer over one input: each reads the input times its branch’s input
multiplier, under the layer’s call, and their outputs, each times its output multiplier, are summed
in the branches’ order (ParallelMixer). Every branch holds its own parameters, its own decode cache
and its own routers under its name. A prediction depth’s phase reaches the branches that read it
(MixerBase.reads_phase), the rest running their ordinary call; a routing replay reaches the branches
that route (RoutesExperts), each its own sites’ records by their paths below it (routed_sites,
dew.nn.protocols.RoutesBySite), or the one routed site a layer’s single record.
branches: tuple[tuple[str, Callable[..., nn.Module], float, float, bool], ...]-
Each branch’s name, its mixer’s factory, its input and output multipliers, and whether it reads a prediction phase.
reads: frozenset[str]-
The
MIXINGfields every branch reads (dew.nn.backbones.decoder_block.ReadsMixing; one that declares none reads none): the call carries each to every branch, so a field one branch would not read is one the block refuses, rather than a restriction that branch would drop. routes_experts: bool-
Whether a branch routes over experts (
dew.nn.protocols.RoutesExperts), so the layer’s routing replay reaches it. routed_sites: frozenset[str]-
Its routed sites’ paths below it: a routed branch’s name, or a composed branch’s sites below its name (
dew.nn.protocols.RoutesBySite). reads_train: bool-
Whether a branch takes the train flag (
dew.nn.protocols.ReadsTrain).
ParallelMixers.setup
Section titled “ParallelMixers.setup”def setup()ParallelMixers.model_flops
Section titled “ParallelMixers.model_flops”def model_flops(lengths) -> floatIts branches’ work; the scalings and the sum are elementwise.
ParallelMixers.parameter_flops
Section titled “ParallelMixers.parameter_flops”def parameter_flops(lengths) -> floatIts branches’ learned maps.
ParallelMixers.projection_groups
Section titled “ParallelMixers.projection_groups”def projection_groups() -> tuple[ProjectionGroup, ...]The packed groups its branches declare (dew.nn.protocols.ProjectionSites).
ParallelMixer
Section titled “ParallelMixer”class ParallelMixer(branches: tuple[ParallelBranch, ...] = ())Several mixer kinds side by side in one layer (ParallelMixers), each ParallelBranch its own
mixer under its name, at the layer’s geometry: a parallel hybrid, Falcon-H1’s Mamba-2 beside its
attention. What a branch keeps, reads or refuses, the layer keeps, reads or refuses: its cache, its
layer’s index (reads_depth, which the call hands every branch), its remat marks, its mixed serving
step, its generation, a prediction depth’s phase (reads_phase), and its routing replay, by its
name.
ParallelMixer.publishes
Section titled “ParallelMixer.publishes”def publishes(kv_shared: bool) -> boolParallelMixer.marks
Section titled “ParallelMixer.marks”def marks() -> tuple[str, ...] | NoneThe names any branch marks; every name where one marks every name.
ParallelMixer.mixed_refusal
Section titled “ParallelMixer.mixed_refusal”def mixed_refusal(ctx: MixerContext) -> str | NoneParallelMixer.generation_refusal
Section titled “ParallelMixer.generation_refusal”def generation_refusal() -> str | NoneParallelMixer.build
Section titled “ParallelMixer.build”def build(ctx: MixerContext) -> Callable[..., nn.Module]