Skip to content

dew.nn.sharding

Where a parameter is split across devices, declared where the module creates it.

A module draws each parameter it names the logical axes of in Flax’s box, outermost dimension first (nn.with_logical_partitioning, declared):

self.param("kernel", declared(init, ("embed", "heads")), shape)
Dense(features, kernel_axes=("attention", "embed"))

so an init returns its parameters boxed, as Flax’s partitioning reads them. A run reads its objective’s draw once, traced for its shapes alone, into one table by each parameter’s path (boxed_axes, Objective.parameter_axes), and holds its trees as plain arrays past where they enter (unboxed). The Layout reads that table when it places a train state, and Muon reads it when it picks a parameter’s matrix axes. The optimizer’s moments and the EMA copy have paths that end in their parameter’s path, so one declaration covers them as well.

A parameter that no declaration names (a convolution, a state matrix, a projection with no side worth naming) gets the shape heuristic, which places it on its largest divisible axis.

This module also defines the functions that read the mesh in context: pipeline_stages for a pipeline’s stage count, microbatches for the schedule the trainer sets in context around its compiled step, in_pipeline_stage for whether a stage’s body is tracing, whose vmap holds the stage axis, sequence_shards for how many ways attention and the Mamba-2 mixer split a sequence, and manual_map for their shard_maps, whose rows split over the axes dew.mesh.row_axes names.

The mesh’s axis names, the rule table that maps the logical names onto them (dew.mesh.DEFAULT_RULES, for parameters and activations alike) and the spec it gives an array (dew.mesh.logical_spec) are dew.mesh’s, below the models and the data; constrain applies that spec to an activation.

NameSummary
ADAPTER_RANKThe logical axis of a low-rank adapter’s rank (LOW_RANK_FACTORS), which no rule places.
RESIDUALThe logical axes of a [batch, length, width] activation between sublayers, which is also each sublayer’s input.
LinkA mesh axis’s interconnect, one device’s dense bf16 peak, and whether a projection split over the axis.
ScheduleThe microbatch count a step feeds the stage axis, and whether a model in the step ran a pipeline.
layers_ofThe stored layers’ paths a leaf at path stands for, and how many of its leading dimensions stack them: a stage-local run’s layers (stacked_layers), each optimizer rule reading the leaf as those layers, or the leaf itself, stacked over none.

attribute source

ADAPTER_RANK = 'adapter_rank'

The logical axis of a low-rank adapter’s rank (LOW_RANK_FACTORS), which no rule places.

attribute source

RESIDUAL: LogicalAxes = ('activation_batch', 'activation_length', 'activation_embed')

The logical axes of a [batch, length, width] activation between sublayers, which is also each sublayer’s input. Under a tensor axis the width stays whole, so every tensor shard reads every row it computes its share of the width for.

dataclass source

class Link(
bytes_per_second: float | None,
flops_per_second: float | None,
spread: bool = False,
)

A mesh axis’s interconnect, one device’s dense bf16 peak, and whether a projection split over the axis.

The trainer measures the interconnect when it places a step (dew.training.distributed.link_bandwidth), or takes it as stated (Trainer(links=...)). down_projection and split_positions set spread when they decide to split a projection in the step over the axis.

bytes_per_second: float | None

The bytes one device receives per second in an all-gather over the axis: (N - 1) / N of the result, divided by the time the gather took. None where nothing was measured or stated: on a device the peak table does not name, a CPU among them.

flops_per_second: float | None

One device’s dense bf16 peak in FLOPs per second, or None for hardware Dew’s target table does not name (dew.nn.kernels.gpu_target(...).bf16_peak).

dataclass source

class Schedule(count: int | None, pipelined: bool = False)

The microbatch count a step feeds the stage axis, and whether a model in the step ran a pipeline.

microbatches sets pipelined when the pipeline reads the count.

function source

def layers_of(path: tuple) -> tuple[tuple[tuple, ...], int]

The stored layers’ paths a leaf at path stands for, and how many of its leading dimensions stack them: a stage-local run’s layers (stacked_layers), each optimizer rule reading the leaf as those layers, or the leaf itself, stacked over none.