Skip to content

dew.distributed

What the processes of a pool do together on the host: agree a phase or a value before going on, so a failure on one rank reaches every rank rather than leaving the rest inside a collective (agreed, agree_process_phase, agreed_same), and bring a global tree to every rank or the first (collective_host), one rank’s value to all (broadcast_from_process_zero) or every rank’s to each (from_every_process). A custom evaluation or training hook that reads data on the host calls these; Dew’s own layers read the same engine (dew._coordination). Where a run’s arrays live, its mesh (MeshSpec) and its layout over it (Layout), is dew.distributed.placement’s.

NameSummary
LayoutDecides how a train state is placed on a mesh. Documented in dew.distributed.placement.
MeshSpecDevice counts for the mesh’s sharding axes; the data axis takes the devices left over. Documented in dew.distributed.placement.
PeerFailureRaised when another process failed at a phase agreement that this process passed.
agree_process_phasePropagate host errors, then count ranks declaring availability.
agreedRun operation on every rank, then agree on the outcome before going on.
agreed_sameWhat each rank finds for itself, refused when the ranks differ.
agreement_patienceBound how long a rank that reached an agreement waits on the host for the rest, one day by default: a pool whose host phases are short can learn sooner of a rank stalled before one.
broadcast_from_process_zeroA JSON-encodable value broadcast from rank zero to every rank (from_every_process).
collective_hostMaterialize an evaluation tree on every rank with transfer consensus.
from_every_processEvery rank’s JSON-encodable value, in rank order, in one round of the coordination service (_round).

class source

class PeerFailure(RuntimeError)

Raised when another process failed at a phase agreement that this process passed.

function source

def agree_process_phase(
error: BaseException | None,
*,
phase: str,
available: bool = True,
) -> int

Propagate host errors, then count ranks declaring availability.

All live ranks must reach this boundary. It cannot rescue a blocked device collective: a rank that failed between steps meets peers still inside the next step’s collectives, which wait for it for ever on a GPU. So a failing rank first publishes its error (publish_failure), which ends the pool within FAILURE_GRACE_SECONDS unless the agreement completes and withdraws it. Errors take priority over unavailable input.

function source

def agreed(phase: str, operation: Callable[[], T]) -> T

Run operation on every rank, then agree on the outcome before going on.

A rank that fails reports its error at the agreement point instead of raising alone, so its peers hear about it there rather than hanging at the next collective. The peers raise PeerFailure; the failing rank re-raises its own error. On one process this is a plain call.

function source

def agreed_same(phase: str, find: Callable[[], T]) -> T

What each rank finds for itself, refused when the ranks differ.

For what every process reads or looks up on its own and the pool must share, such as its data, its newest checkpoint or the plan of a gather. find returns a JSON value, and the ranks compare a digest of it, so a large one costs no more to agree than a small one. A failure in find is agreed as in agreed; a rank that found something else than process 0 is refused on every rank, both values named, which takes a second round. On one process this is a plain call.

function source

def agreement_patience(seconds: float) -> None

Bound how long a rank that reached an agreement waits on the host for the rest, one day by default: a pool whose host phases are short can learn sooner of a rank stalled before one.

function source

def broadcast_from_process_zero(value)

A JSON-encodable value broadcast from rank zero to every rank (from_every_process).

function source

def collective_host(
value: T,
*,
phase: str,
held_by: Literal['every', 'first'] = 'every',
) -> T | None

Materialize an evaluation tree on every rank with transfer consensus.

All ranks must call this, even for entirely local trees. Every leaf is waited on first, so a computation that failed on a rank reports at the preflight rather than inside a gather collective. Ranks then agree the ordered global gather plan, and gather global leaves in groups (GATHER_BYTES), agreeing each group’s outcome before the next starts. Local-only leaves may differ across ranks. A device failure inside an in-flight collective still needs runtime termination. With held_by “first” only process 0 copies the tree home and returns it; the others take part in every computation and agreement and return None.

function source

def from_every_process(value) -> list

Every rank’s JSON-encodable value, in rank order, in one round of the coordination service (_round). For the small host values a pool shares; a device array goes through a collective.