dew.nn.sharding
Where a parameter is split across devices, declared where the module creates it.
A module draws each parameter it names the logical axes of in Flax’s box,
outermost dimension first (nn.with_logical_partitioning, declared):
self.param("kernel", declared(init, ("embed", "heads")), shape)Dense(features, kernel_axes=("attention", "embed"))so an init returns its parameters boxed, as Flax’s partitioning reads them.
A run reads its objective’s draw once, traced for its shapes alone, into one
table by each parameter’s path (boxed_axes, Objective.parameter_axes),
and holds its trees as plain arrays past where they enter (unboxed). The
Layout reads that table when it places a train state, and Muon reads it
when it picks a parameter’s matrix axes. The optimizer’s moments and the EMA
copy have paths that end in their parameter’s path, so one declaration covers
them as well.
A parameter that no declaration names (a convolution, a state matrix, a projection with no side worth naming) gets the shape heuristic, which places it on its largest divisible axis.
This module also defines the functions that read the mesh in context:
pipeline_stages for a pipeline’s stage count, microbatches for the
schedule the trainer sets in context around its compiled step,
in_pipeline_stage for whether a stage’s body is tracing, whose vmap holds
the stage axis,
sequence_shards for how many ways attention and the Mamba-2 mixer split a
sequence, and manual_map for their shard_maps, whose rows split over the
axes dew.mesh.row_axes names.
The mesh’s axis names, the rule table that maps the logical names onto them
(dew.mesh.DEFAULT_RULES, for parameters and activations alike) and the spec
it gives an array (dew.mesh.logical_spec) are dew.mesh’s, below the models
and the data; constrain applies that spec to an activation.
| Name | Summary |
|---|---|
ADAPTER_RANK | The logical axis of a low-rank adapter’s rank (LOW_RANK_FACTORS), which no rule places. |
RESIDUAL | The logical axes of a [batch, length, width] activation between sublayers, which is also each sublayer’s input. |
Link | A mesh axis’s interconnect, one device’s dense bf16 peak, and whether a projection split over the axis. |
Schedule | The microbatch count a step feeds the stage axis, and whether a model in the step ran a pipeline. |
layers_of | The stored layers’ paths a leaf at path stands for, and how many of its leading dimensions stack them: a stage-local run’s layers (stacked_layers), each optimizer rule reading the leaf as those layers, or the leaf itself, stacked over none. |
ADAPTER_RANK
Section titled “ADAPTER_RANK”ADAPTER_RANK = 'adapter_rank'The logical axis of a low-rank adapter’s rank (LOW_RANK_FACTORS), which no rule places.
RESIDUAL
Section titled “RESIDUAL”RESIDUAL: LogicalAxes = ('activation_batch', 'activation_length', 'activation_embed')The logical axes of a [batch, length, width] activation between sublayers,
which is also each sublayer’s input. Under a tensor axis the width stays
whole, so every tensor shard reads every row it computes its share of the
width for.
class Link( bytes_per_second: float | None, flops_per_second: float | None, spread: bool = False,)A mesh axis’s interconnect, one device’s dense bf16 peak, and whether a projection split over the axis.
The trainer measures the interconnect when it places a step
(dew.training.distributed.link_bandwidth), or takes it as stated
(Trainer(links=...)). down_projection and
split_positions set spread when they decide to split a projection in
the step over the axis.
bytes_per_second: float | None-
The bytes one device receives per second in an all-gather over the axis: (N - 1) / N of the result, divided by the time the gather took. None where nothing was measured or stated: on a device the peak table does not name, a CPU among them.
flops_per_second: float | None-
One device’s dense bf16 peak in FLOPs per second, or None for hardware Dew’s target table does not name (
dew.nn.kernels.gpu_target(...).bf16_peak).
Schedule
Section titled “Schedule”class Schedule(count: int | None, pipelined: bool = False)The microbatch count a step feeds the stage axis, and whether a model in the step ran a pipeline.
microbatches sets pipelined when the pipeline reads the count.
layers_of
Section titled “layers_of”def layers_of(path: tuple) -> tuple[tuple[tuple, ...], int]The stored layers’ paths a leaf at path stands for, and how many of its leading dimensions stack
them: a stage-local run’s layers (stacked_layers), each optimizer rule reading the leaf as those
layers, or the leaf itself, stacked over none.