Skip to content

dew.mesh

The mesh facts every layer reads, below the models and the data alike: the names of a mesh’s axes (MESH_AXES), the ones a batch’s rows split over (BATCH_AXES), the default table of rules that places each logical axis on them (DEFAULT_RULES, or a layout’s in flax’s context, axis_rules), the spec those rules give an array (logical_spec), which axes a batch’s rows split over on a mesh (row_axes, batch_axes), and the refusal of a layout that does not run (LayoutRefused). The models’ layout (dew.nn.sharding) constrains and maps their arrays by these, and a run’s data shares (dew.data.dataset.DataPartition.of) read the rows’ axes here.

NameSummary
BATCH_AXESThe mesh axes a batch’s rows split over, in mesh order.
DATA_AXIS
DEFAULT_RULES
DILOCO_AXISThe axis of DiLoCo’s replica groups (dew.training.diloco), outermost: each group trains its own inner copy of the model on its own rows, every axis below it inside the group, and the groups meet only when the outer model averages them.
EXPERT_AXIS
FSDP_AXIS
MESH_AXESEvery mesh’s axes, in the order MeshSpec.build lays them out.
SEQUENCE_AXIS
STAGE_AXISThe axes of a mesh MeshSpec.build builds, plus the stage axis a pipeline mesh adds.
TENSOR_AXIS
LayoutRefusedRaised for a mesh layout that a model, an objective or a device set does not run, by design.
LogicalAxes
LogicalAxisRules
MeshAxes
axis_rulesThe rules in context, flax.linen.logical_axis_rules’s, where the trainer puts its layout’s; the default table outside one.
batch_axesThe axes of mesh a batch’s rows split over: DiLoCo’s replica groups first where the mesh has more than one, each group’s rows its own, then every axis activation_batch takes, each of which divides mesh.size rows.
logical_specReturn the spec that rules give, on mesh, an array of shape whose dimensions axes names.
mesh_axesRead one entry of a spec or a rule as the mesh axes it names.
row_axesThe mesh axes a shard_map over the sequence axis splits batch rows over: the ones activation_batch takes for batch rows on mesh, the mesh in context by default (logical_spec).

attribute source

BATCH_AXES = (DATA_AXIS, EXPERT_AXIS, FSDP_AXIS)

The mesh axes a batch’s rows split over, in mesh order. Sequence holds a slice of the positions, stage the pipeline’s stages that hand one batch’s microbatches along, and tensor the widths of Megatron’s split, so every tensor shard computes its share of the width for every row.

attribute source

DATA_AXIS = 'data'

attribute source

DEFAULT_RULES: LogicalAxisRules

attribute source

DILOCO_AXIS = 'diloco'

The axis of DiLoCo’s replica groups (dew.training.diloco), outermost: each group trains its own inner copy of the model on its own rows, every axis below it inside the group, and the groups meet only when the outer model averages them. No model’s rule names it: a step’s model runs inside one group.

attribute source

EXPERT_AXIS = 'expert'

attribute source

FSDP_AXIS = 'fsdp'

attribute source

MESH_AXES = (DILOCO_AXIS, DATA_AXIS, EXPERT_AXIS, FSDP_AXIS, TENSOR_AXIS, SEQUENCE_AXIS, STAGE_AXIS)

Every mesh’s axes, in the order MeshSpec.build lays them out.

attribute source

SEQUENCE_AXIS = 'sequence'

attribute source

STAGE_AXIS = 'stage'

The axes of a mesh MeshSpec.build builds, plus the stage axis a pipeline mesh adds. They are named here because the attention seam and the decoder read them off the mesh in context: the sequence axis attention splits its queries over, and the tensor and stage axes, which hold a width and a pipeline stage and never a row.

attribute source

TENSOR_AXIS = 'tensor'

class source

class LayoutRefused(ValueError)

Raised for a mesh layout that a model, an objective or a device set does not run, by design.

The message names the axis, says why it cannot hold what it would split, and gives a layout that runs. Any other error on a layout is a defect. It is a ValueError, so a caller that catches ValueError catches it too.

attribute source

attribute source

attribute source

function source

def axis_rules() -> LogicalAxisRules

The rules in context, flax.linen.logical_axis_rules’s, where the trainer puts its layout’s; the default table outside one.

function source

def batch_axes(mesh: jax.sharding.Mesh | jax.sharding.AbstractMesh) -> tuple[str, ...]

The axes of mesh a batch’s rows split over: DiLoCo’s replica groups first where the mesh has more than one, each group’s rows its own, then every axis activation_batch takes, each of which divides mesh.size rows.

function source

def logical_spec(
axes: LogicalAxes,
shape: tuple[int, ...],
*,
rules: LogicalAxisRules | None = None,
mesh: jax.sharding.Mesh | jax.sharding.AbstractMesh | None = None,
) -> jax.sharding.PartitionSpec

Return the spec that rules give, on mesh, an array of shape whose dimensions axes names.

rules and mesh default to the ones in context. A mesh axis of size 1 shards nothing, and a manual axis belongs to the shard_map in context, so both are dropped from the rules before they are read. Such an axis claims no dimension, and a name whose rule named only such axes takes its next rule. With no tensor axis, ‘mlp’ passes its tensor rule to fsdp. A rule whose axes do not divide its dimension evenly cannot split it, so that rule is set aside for this array: the name takes its next rule, or none, and the mesh axis goes to the next dimension that names it. For example, an odd vocabulary shards the embedding on its width and keeps the table in the layout. fsdp, placed by a name whose first rule names it alone, then moves to the leading dimension of such a name that no other axis took and that it divides (DEFAULT_RULES).

function source

def mesh_axes(assignment: MeshAxes) -> tuple[str, ...]

Read one entry of a spec or a rule as the mesh axes it names.

function source

def row_axes(
batch: int,
*,
mesh: jax.sharding.Mesh | jax.sharding.AbstractMesh | None = None,
) -> tuple[str, ...]

The mesh axes a shard_map over the sequence axis splits batch rows over: the ones activation_batch takes for batch rows on mesh, the mesh in context by default (logical_spec). A batch too small for them is computed alike on those axes’ shards, the way GSPMD replicates a dimension it cannot split.