Skip to content

dew.clusters

The scheduler a run’s processes start under, and the place each one takes in the run’s process pool.

A run across hosts is one JAX process pool: jax.distributed.initialize takes the coordinator’s address, the pool’s process count, this process’s rank and the accelerators it takes on its host. A scheduler that starts the processes knows them: Slurm, Open MPI, a Kubernetes indexed Job, a Cloud TPU’s runtime, or dew launch, which writes them into each process’s environment (Launched). A ClusterEnvironment says how a process joins under one scheduler; prepare_process(cluster=...) joins the pool it describes, or, given none, the first of ENVIRONMENTS this process runs under (detected_environment). A scheduler Dew does not know is a class of the user’s own:

@dataclass(frozen=True)
class MyScheduler:
name = "my scheduler"
def detected(self) -> bool:
return "MY_RANK" in os.environ
def joining(self) -> Joining:
return Joining(os.environ["MY_COORDINATOR"], int(os.environ["MY_WORLD"]),
int(os.environ["MY_RANK"]), detection="deactivate")
prepare_process(cluster=MyScheduler())

A scheduler JAX detects itself joins by JAX’s own detection of it (Joining.detection, its cluster_detection_method), which derives every fact JAX’s way: a Slurm step’s coordinator from its node list, a multislice TPU’s global ranks, the local devices JAX_LOCAL_DEVICE_IDS names. Dew reads a scheduler’s presence as JAX’s detection does (jax/_src/clusters, jax-ml/jax 19a48d1de), so a process Dew finds under one JAX finds under it too, and its place where Dew’s own decisions need it (Scheduler.place: a Slurm step of one task, the pool dew launch runs in place). This module imports only the standard library, so dew launch reads it without JAX.

NameSummary
ENVIRONMENTSEvery environment prepare_process detects, dew launch’s first: a pool it declared is placed whole.
SCHEDULERSThe schedulers Dew reads, in JAX’s own detection order (jax/_src/clusters/init.py): Open MPI’s ranks before a Slurm step’s tasks, which mpirun inside a Slurm allocation sets both of.
CloudTPUA Cloud TPU’s workers, which JAX’s detection places whole, multislice ranks among them: on GKE a TPU VM whose worker list the environment states, joined by name (GkeTpuCluster); any other TPU VM, TPU_SKIP_MDS_QUERY unset, by JAX’s own order of detections, which asks GCE’s metadata server and, where it names no worker, goes on to the next (a Kubernetes Job’s), so Dew asks no server itself.
ClusterEnvironmentA scheduler that starts a run’s processes and tells each its place: detected where this process runs under it, joining how it joins the pool.
JoiningWhat a process passes jax.distributed.initialize to join its pool: each fact its environment states, None where it states none, and detection, JAX’s cluster_detection_method: a scheduler JAX detects by its name (“slurm”, “ompi”, “gcetpu”, “gketpu”), which fills every fact left None, or “deactivate” where the environment states the pool whole, so a scheduler around it (a Slurm allocation dew launch runs in) places the process no further; None is JAX’s own order of detection.
KubernetesThe pods of a Kubernetes Job, where the kubernetes package is installed for JAX’s detection to read the cluster’s API (K8sCluster), which refuses a Job whose completions are not indexed.
LaunchedA pool dew launch started: it writes each process’s place into its environment (environ_of), the whole of it, so JAX detects nothing more.
OpenMPIProcesses mpirun started: Open MPI’s legacy launcher, which states its ORTE URI, or Open MPI 5’s, which states its ranks and a major version of 5 or more, or, stating no version, no ORTE URI (OmpiCluster.is_env_present, _is_ompi_5); the coordinator JAX’s detection reads off the launcher.
SchedulerA scheduler Dew detects itself (SCHEDULERS), whose place states the pool’s size, this process’s rank and its accelerators where the scheduler tells them without JAX: what Dew’s own decisions read, dew launch’s running in place among them, none of which reaches JAX.
SlurmThe tasks of a Slurm step: SLURM_NTASKS of them, this one SLURM_PROCID, taking the accelerator at SLURM_LOCALID, as JAX places them; the coordinator JAX’s detection names after the step’s first node.
detected_environmentThe first of environments this process runs under, reading environ (the process’s own where None), or None.

attribute source

ENVIRONMENTS: tuple[Reader, ...] = (Launched, *SCHEDULERS)

Every environment prepare_process detects, dew launch’s first: a pool it declared is placed whole.

attribute source

SCHEDULERS: tuple[Reader, ...] = (OpenMPI, Slurm, CloudTPU, Kubernetes)

The schedulers Dew reads, in JAX’s own detection order (jax/_src/clusters/init.py): Open MPI’s ranks before a Slurm step’s tasks, which mpirun inside a Slurm allocation sets both of.

dataclass source

class CloudTPU(environ: Mapping[str, str] | None = None)

A Cloud TPU’s workers, which JAX’s detection places whole, multislice ranks among them: on GKE a TPU VM whose worker list the environment states, joined by name (GkeTpuCluster); any other TPU VM, TPU_SKIP_MDS_QUERY unset, by JAX’s own order of detections, which asks GCE’s metadata server and, where it names no worker, goes on to the next (a Kubernetes Job’s), so Dew asks no server itself. Its place is JAX’s to read as it joins: Dew decides nothing by it before, so it states none, and dew launch runs every worker’s program in place.

def detected() -> bool
def joining() -> Joining
def place() -> Joining

class source

class ClusterEnvironment(Protocol)

A scheduler that starts a run’s processes and tells each its place: detected where this process runs under it, joining how it joins the pool.

def detected() -> bool
def joining() -> Joining

dataclass source

class Joining(
coordinator_address: str | None = None,
process_count: int | None = None,
process_id: int | None = None,
local_device_ids: tuple[int, ...] | None = None,
detection: str | None = None,
)

What a process passes jax.distributed.initialize to join its pool: each fact its environment states, None where it states none, and detection, JAX’s cluster_detection_method: a scheduler JAX detects by its name (“slurm”, “ompi”, “gcetpu”, “gketpu”), which fills every fact left None, or “deactivate” where the environment states the pool whole, so a scheduler around it (a Slurm allocation dew launch runs in) places the process no further; None is JAX’s own order of detection.

dataclass source

class Kubernetes(environ: Mapping[str, str] | None = None)

The pods of a Kubernetes Job, where the kubernetes package is installed for JAX’s detection to read the cluster’s API (K8sCluster), which refuses a Job whose completions are not indexed.

def detected() -> bool
def joining() -> Joining
def place() -> Joining

dataclass source

class Launched(environ: Mapping[str, str] | None = None)

A pool dew launch started: it writes each process’s place into its environment (environ_of), the whole of it, so JAX detects nothing more.

def detected() -> bool
def joining() -> Joining
def place() -> Joining
def environ_of(
coordinator_address: str,
process_count: int,
process_id: int | None = None,
local_device_ids: Sequence[int] | None = None,
) -> dict[str, str]

The variables dew launch writes into a process for Launched to read: the pool’s, and this process’s rank and accelerators where given.

dataclass source

class OpenMPI(environ: Mapping[str, str] | None = None)

Processes mpirun started: Open MPI’s legacy launcher, which states its ORTE URI, or Open MPI 5’s, which states its ranks and a major version of 5 or more, or, stating no version, no ORTE URI (OmpiCluster.is_env_present, _is_ompi_5); the coordinator JAX’s detection reads off the launcher.

def detected() -> bool
def joining() -> Joining
def place() -> Joining

class source

class Scheduler(ClusterEnvironment, Protocol)

A scheduler Dew detects itself (SCHEDULERS), whose place states the pool’s size, this process’s rank and its accelerators where the scheduler tells them without JAX: what Dew’s own decisions read, dew launch’s running in place among them, none of which reaches JAX.

def place() -> Joining

dataclass source

class Slurm(environ: Mapping[str, str] | None = None)

The tasks of a Slurm step: SLURM_NTASKS of them, this one SLURM_PROCID, taking the accelerator at SLURM_LOCALID, as JAX places them; the coordinator JAX’s detection names after the step’s first node.

def detected() -> bool
def joining() -> Joining
def place() -> Joining

function source

def detected_environment(
environments: Sequence[Reader] = ENVIRONMENTS,
environ: Mapping[str, str] | None = None,
) -> Scheduler | None

The first of environments this process runs under, reading environ (the process’s own where None), or None.