dew.clusters
The scheduler a run’s processes start under, and the place each one takes in the run’s process pool.
A run across hosts is one JAX process pool: jax.distributed.initialize takes the coordinator’s address, the
pool’s process count, this process’s rank and the accelerators it takes on its host. A scheduler that starts
the processes knows them: Slurm, Open MPI, a Kubernetes indexed Job, a Cloud TPU’s runtime, or dew launch,
which writes them into each process’s environment (Launched). A ClusterEnvironment says how a process
joins under one scheduler; prepare_process(cluster=...) joins the pool it describes, or, given none, the
first of ENVIRONMENTS this process runs under (detected_environment). A scheduler Dew does not know is a
class of the user’s own:
@dataclass(frozen=True)class MyScheduler: name = "my scheduler"
def detected(self) -> bool: return "MY_RANK" in os.environ
def joining(self) -> Joining: return Joining(os.environ["MY_COORDINATOR"], int(os.environ["MY_WORLD"]), int(os.environ["MY_RANK"]), detection="deactivate")
prepare_process(cluster=MyScheduler())A scheduler JAX detects itself joins by JAX’s own detection of it (Joining.detection, its
cluster_detection_method), which derives every fact JAX’s way: a Slurm step’s coordinator from its node
list, a multislice TPU’s global ranks, the local devices JAX_LOCAL_DEVICE_IDS names. Dew reads a
scheduler’s presence as JAX’s detection does (jax/_src/clusters, jax-ml/jax 19a48d1de), so a process Dew finds
under one JAX finds under it too, and its place where Dew’s own decisions need it (Scheduler.place: a Slurm
step of one task, the pool dew launch runs in place). This module imports only the standard library, so
dew launch reads it without JAX.
| Name | Summary |
|---|---|
ENVIRONMENTS | Every environment prepare_process detects, dew launch’s first: a pool it declared is placed whole. |
SCHEDULERS | The schedulers Dew reads, in JAX’s own detection order (jax/_src/clusters/init.py): Open MPI’s ranks before a Slurm step’s tasks, which mpirun inside a Slurm allocation sets both of. |
CloudTPU | A Cloud TPU’s workers, which JAX’s detection places whole, multislice ranks among them: on GKE a TPU VM whose worker list the environment states, joined by name (GkeTpuCluster); any other TPU VM, TPU_SKIP_MDS_QUERY unset, by JAX’s own order of detections, which asks GCE’s metadata server and, where it names no worker, goes on to the next (a Kubernetes Job’s), so Dew asks no server itself. |
ClusterEnvironment | A scheduler that starts a run’s processes and tells each its place: detected where this process runs under it, joining how it joins the pool. |
Joining | What a process passes jax.distributed.initialize to join its pool: each fact its environment states, None where it states none, and detection, JAX’s cluster_detection_method: a scheduler JAX detects by its name (“slurm”, “ompi”, “gcetpu”, “gketpu”), which fills every fact left None, or “deactivate” where the environment states the pool whole, so a scheduler around it (a Slurm allocation dew launch runs in) places the process no further; None is JAX’s own order of detection. |
Kubernetes | The pods of a Kubernetes Job, where the kubernetes package is installed for JAX’s detection to read the cluster’s API (K8sCluster), which refuses a Job whose completions are not indexed. |
Launched | A pool dew launch started: it writes each process’s place into its environment (environ_of), the whole of it, so JAX detects nothing more. |
OpenMPI | Processes mpirun started: Open MPI’s legacy launcher, which states its ORTE URI, or Open MPI 5’s, which states its ranks and a major version of 5 or more, or, stating no version, no ORTE URI (OmpiCluster.is_env_present, _is_ompi_5); the coordinator JAX’s detection reads off the launcher. |
Scheduler | A scheduler Dew detects itself (SCHEDULERS), whose place states the pool’s size, this process’s rank and its accelerators where the scheduler tells them without JAX: what Dew’s own decisions read, dew launch’s running in place among them, none of which reaches JAX. |
Slurm | The tasks of a Slurm step: SLURM_NTASKS of them, this one SLURM_PROCID, taking the accelerator at SLURM_LOCALID, as JAX places them; the coordinator JAX’s detection names after the step’s first node. |
detected_environment | The first of environments this process runs under, reading environ (the process’s own where None), or None. |
ENVIRONMENTS
Section titled “ENVIRONMENTS”ENVIRONMENTS: tuple[Reader, ...] = (Launched, *SCHEDULERS)Every environment prepare_process detects, dew launch’s first: a pool it declared is placed whole.
SCHEDULERS
Section titled “SCHEDULERS”SCHEDULERS: tuple[Reader, ...] = (OpenMPI, Slurm, CloudTPU, Kubernetes)The schedulers Dew reads, in JAX’s own detection order (jax/_src/clusters/init.py): Open MPI’s ranks
before a Slurm step’s tasks, which mpirun inside a Slurm allocation sets both of.
CloudTPU
Section titled “CloudTPU”class CloudTPU(environ: Mapping[str, str] | None = None)A Cloud TPU’s workers, which JAX’s detection places whole, multislice ranks among them: on GKE a TPU
VM whose worker list the environment states, joined by name (GkeTpuCluster); any other TPU VM,
TPU_SKIP_MDS_QUERY unset, by JAX’s own order of detections, which asks GCE’s metadata server and, where
it names no worker, goes on to the next (a Kubernetes Job’s), so Dew asks no server itself. Its place is
JAX’s to read as it joins: Dew decides nothing by it before, so it states none, and dew launch runs
every worker’s program in place.
CloudTPU.detected
Section titled “CloudTPU.detected”def detected() -> boolCloudTPU.joining
Section titled “CloudTPU.joining”def joining() -> JoiningCloudTPU.place
Section titled “CloudTPU.place”def place() -> JoiningClusterEnvironment
Section titled “ClusterEnvironment”class ClusterEnvironment(Protocol)A scheduler that starts a run’s processes and tells each its place: detected where this process
runs under it, joining how it joins the pool.
ClusterEnvironment.detected
Section titled “ClusterEnvironment.detected”def detected() -> boolClusterEnvironment.joining
Section titled “ClusterEnvironment.joining”def joining() -> JoiningJoining
Section titled “Joining”class Joining( coordinator_address: str | None = None, process_count: int | None = None, process_id: int | None = None, local_device_ids: tuple[int, ...] | None = None, detection: str | None = None,)What a process passes jax.distributed.initialize to join its pool: each fact its environment
states, None where it states none, and detection, JAX’s cluster_detection_method: a scheduler JAX
detects by its name (“slurm”, “ompi”, “gcetpu”, “gketpu”), which fills every fact left None, or
“deactivate” where the environment states the pool whole, so a scheduler around it (a Slurm
allocation dew launch runs in) places the process no further; None is JAX’s own order of detection.
Kubernetes
Section titled “Kubernetes”class Kubernetes(environ: Mapping[str, str] | None = None)The pods of a Kubernetes Job, where the kubernetes package is installed for JAX’s detection to read
the cluster’s API (K8sCluster), which refuses a Job whose completions are not indexed.
Kubernetes.detected
Section titled “Kubernetes.detected”def detected() -> boolKubernetes.joining
Section titled “Kubernetes.joining”def joining() -> JoiningKubernetes.place
Section titled “Kubernetes.place”def place() -> JoiningLaunched
Section titled “Launched”class Launched(environ: Mapping[str, str] | None = None)A pool dew launch started: it writes each process’s place into its environment (environ_of),
the whole of it, so JAX detects nothing more.
Launched.detected
Section titled “Launched.detected”def detected() -> boolLaunched.joining
Section titled “Launched.joining”def joining() -> JoiningLaunched.place
Section titled “Launched.place”def place() -> JoiningLaunched.environ_of
Section titled “Launched.environ_of”def environ_of( coordinator_address: str, process_count: int, process_id: int | None = None, local_device_ids: Sequence[int] | None = None,) -> dict[str, str]The variables dew launch writes into a process for Launched to read: the pool’s, and this
process’s rank and accelerators where given.
OpenMPI
Section titled “OpenMPI”class OpenMPI(environ: Mapping[str, str] | None = None)Processes mpirun started: Open MPI’s legacy launcher, which states its ORTE URI, or Open MPI 5’s,
which states its ranks and a major version of 5 or more, or, stating no version, no ORTE URI
(OmpiCluster.is_env_present, _is_ompi_5); the coordinator JAX’s detection reads off the launcher.
OpenMPI.detected
Section titled “OpenMPI.detected”def detected() -> boolOpenMPI.joining
Section titled “OpenMPI.joining”def joining() -> JoiningOpenMPI.place
Section titled “OpenMPI.place”def place() -> JoiningScheduler
Section titled “Scheduler”class Scheduler(ClusterEnvironment, Protocol)A scheduler Dew detects itself (SCHEDULERS), whose place states the pool’s size, this process’s
rank and its accelerators where the scheduler tells them without JAX: what Dew’s own decisions read,
dew launch’s running in place among them, none of which reaches JAX.
Scheduler.place
Section titled “Scheduler.place”def place() -> Joiningclass Slurm(environ: Mapping[str, str] | None = None)The tasks of a Slurm step: SLURM_NTASKS of them, this one SLURM_PROCID, taking the accelerator at SLURM_LOCALID, as JAX places them; the coordinator JAX’s detection names after the step’s first node.
Slurm.detected
Section titled “Slurm.detected”def detected() -> boolSlurm.joining
Section titled “Slurm.joining”def joining() -> JoiningSlurm.place
Section titled “Slurm.place”def place() -> Joiningdetected_environment
Section titled “detected_environment”def detected_environment( environments: Sequence[Reader] = ENVIRONMENTS, environ: Mapping[str, str] | None = None,) -> Scheduler | NoneThe first of environments this process runs under, reading environ (the process’s own where
None), or None.