dew.mesh
The mesh facts every layer reads, below the models and the data alike: the names of a mesh’s axes
(MESH_AXES), the ones a batch’s rows split over (BATCH_AXES), the default table of rules that
places each logical axis on them (DEFAULT_RULES, or a layout’s in flax’s context, axis_rules),
the spec those rules give an array (logical_spec), which axes a batch’s rows split over on a mesh
(row_axes, batch_axes), and the refusal of a layout that does not run (LayoutRefused). The
models’ layout (dew.nn.sharding) constrains and maps their arrays by these, and a run’s data shares
(dew.data.dataset.DataPartition.of) read the rows’ axes here.
| Name | Summary |
|---|---|
BATCH_AXES | The mesh axes a batch’s rows split over, in mesh order. |
DATA_AXIS | |
DEFAULT_RULES | |
DILOCO_AXIS | The axis of DiLoCo’s replica groups (dew.training.diloco), outermost: each group trains its own inner copy of the model on its own rows, every axis below it inside the group, and the groups meet only when the outer model averages them. |
EXPERT_AXIS | |
FSDP_AXIS | |
MESH_AXES | Every mesh’s axes, in the order MeshSpec.build lays them out. |
SEQUENCE_AXIS | |
STAGE_AXIS | The axes of a mesh MeshSpec.build builds, plus the stage axis a pipeline mesh adds. |
TENSOR_AXIS | |
LayoutRefused | Raised for a mesh layout that a model, an objective or a device set does not run, by design. |
LogicalAxes | |
LogicalAxisRules | |
MeshAxes | |
axis_rules | The rules in context, flax.linen.logical_axis_rules’s, where the trainer puts its layout’s; the default table outside one. |
batch_axes | The axes of mesh a batch’s rows split over: DiLoCo’s replica groups first where the mesh has more than one, each group’s rows its own, then every axis activation_batch takes, each of which divides mesh.size rows. |
logical_spec | Return the spec that rules give, on mesh, an array of shape whose dimensions axes names. |
mesh_axes | Read one entry of a spec or a rule as the mesh axes it names. |
row_axes | The mesh axes a shard_map over the sequence axis splits batch rows over: the ones activation_batch takes for batch rows on mesh, the mesh in context by default (logical_spec). |
BATCH_AXES
Section titled “BATCH_AXES”BATCH_AXES = (DATA_AXIS, EXPERT_AXIS, FSDP_AXIS)The mesh axes a batch’s rows split over, in mesh order. Sequence holds a slice of the positions, stage the pipeline’s stages that hand one batch’s microbatches along, and tensor the widths of Megatron’s split, so every tensor shard computes its share of the width for every row.
DATA_AXIS
Section titled “DATA_AXIS”DATA_AXIS = 'data'DEFAULT_RULES
Section titled “DEFAULT_RULES”DEFAULT_RULES: LogicalAxisRulesDILOCO_AXIS
Section titled “DILOCO_AXIS”DILOCO_AXIS = 'diloco'The axis of DiLoCo’s replica groups (dew.training.diloco), outermost: each group trains its own
inner copy of the model on its own rows, every axis below it inside the group, and the groups meet only
when the outer model averages them. No model’s rule names it: a step’s model runs inside one group.
EXPERT_AXIS
Section titled “EXPERT_AXIS”EXPERT_AXIS = 'expert'FSDP_AXIS
Section titled “FSDP_AXIS”FSDP_AXIS = 'fsdp'MESH_AXES
Section titled “MESH_AXES”MESH_AXES = (DILOCO_AXIS, DATA_AXIS, EXPERT_AXIS, FSDP_AXIS, TENSOR_AXIS, SEQUENCE_AXIS, STAGE_AXIS)Every mesh’s axes, in the order MeshSpec.build lays them out.
SEQUENCE_AXIS
Section titled “SEQUENCE_AXIS”SEQUENCE_AXIS = 'sequence'STAGE_AXIS
Section titled “STAGE_AXIS”STAGE_AXIS = 'stage'The axes of a mesh MeshSpec.build builds, plus the stage axis a
pipeline mesh adds. They are named here because the attention seam and the
decoder read them off the mesh in context: the sequence axis attention splits
its queries over, and the tensor and stage axes, which hold a width and a
pipeline stage and never a row.
TENSOR_AXIS
Section titled “TENSOR_AXIS”TENSOR_AXIS = 'tensor'LayoutRefused
Section titled “LayoutRefused”class LayoutRefused(ValueError)Raised for a mesh layout that a model, an objective or a device set does not run, by design.
The message names the axis, says why it cannot hold what it would split, and gives a layout that runs. Any other error on a layout is a defect. It is a ValueError, so a caller that catches ValueError catches it too.
LogicalAxes
Section titled “LogicalAxes”LogicalAxisRules
Section titled “LogicalAxisRules”MeshAxes
Section titled “MeshAxes”axis_rules
Section titled “axis_rules”def axis_rules() -> LogicalAxisRulesThe rules in context, flax.linen.logical_axis_rules’s, where the
trainer puts its layout’s; the default table outside one.
batch_axes
Section titled “batch_axes”def batch_axes(mesh: jax.sharding.Mesh | jax.sharding.AbstractMesh) -> tuple[str, ...]The axes of mesh a batch’s rows split over: DiLoCo’s replica groups first where the mesh has
more than one, each group’s rows its own, then every axis activation_batch takes, each of which
divides mesh.size rows.
logical_spec
Section titled “logical_spec”def logical_spec( axes: LogicalAxes, shape: tuple[int, ...], *, rules: LogicalAxisRules | None = None, mesh: jax.sharding.Mesh | jax.sharding.AbstractMesh | None = None,) -> jax.sharding.PartitionSpecReturn the spec that rules give, on mesh, an array of shape whose dimensions axes names.
rules and mesh default to the ones in context. A mesh axis of size 1
shards nothing, and a manual axis belongs to the shard_map in context,
so both are dropped from the rules before they are read. Such an axis
claims no dimension, and a name whose rule named only such axes takes
its next rule. With no tensor axis, ‘mlp’ passes its tensor rule to
fsdp. A rule whose axes do not divide its dimension evenly cannot split
it, so that rule is set aside for this array: the name takes its next
rule, or none, and the mesh axis goes to the next dimension that names
it. For example, an odd vocabulary shards the embedding on its width and
keeps the table in the layout. fsdp, placed by a name whose first rule
names it alone, then moves to the leading dimension of such a name that
no other axis took and that it divides (DEFAULT_RULES).
mesh_axes
Section titled “mesh_axes”def mesh_axes(assignment: MeshAxes) -> tuple[str, ...]Read one entry of a spec or a rule as the mesh axes it names.
row_axes
Section titled “row_axes”def row_axes( batch: int, *, mesh: jax.sharding.Mesh | jax.sharding.AbstractMesh | None = None,) -> tuple[str, ...]The mesh axes a shard_map over the sequence axis splits batch rows
over: the ones activation_batch takes for batch rows on mesh, the
mesh in context by default (logical_spec). A batch too small for them is
computed alike on those axes’ shards, the way GSPMD replicates a dimension
it cannot split.