dew.training.runtime
Process setup that every recipe runs before it builds anything.
Every recipe needs the same setup: the file descriptor limit, the XLA flags,
the compilation cache, the JAX distributed pool, and the environment
variables that wandb and the tokenizers read. So the library does it in
prepare_process, and each recipe calls that once at the top of main().
| Name | Summary |
|---|---|
Preempted | Raised by Trainer.fit when it stops at a preemption notice. |
PreemptionNotice | Reports once a step whether a preemption notice has reached the run, from its creation until close. |
prepare_process | Set the environment variables and XLA flags, raise the soft file descriptor limit, and join the JAX process pool. |
run_timestamp | Return process 0’s wall clock as %Y-%m-%d_%H:%M:%S, on every process. |
unprepared | Which of prepare_process’s options a run states that this process does not carry: a flag of xla_flags XLA_FLAGS did not carry while the backend was closed (flags_set), a compilation cache JAX keeps elsewhere, a pool it has not joined (multi_host=True; False only says never to join one). |
Preempted
Section titled “Preempted”class Preempted(step: int)Raised by Trainer.fit when it stops at a preemption notice.
step is the step it stopped at, and fit has written that step’s
checkpoint and data position before raising. Uncaught, it ends the
program with exit status PREEMPTED_EXIT and no traceback, as SIGTERM
itself would have. Run the same program again to resume from the
checkpoint.
PreemptionNotice
Section titled “PreemptionNotice”class PreemptionNotice()Reports once a step whether a preemption notice has reached the run,
from its creation until close.
A scheduler stops a job by sending SIGTERM and then SIGKILL after a grace
period, such as Slurm’s KillWait, Kubernetes’ termination grace period
or a spot VM’s notice. In a process pool, JAX’s preemption service
receives the SIGTERM; XLA’s notifier replaces the signal handler, so the
process keeps running. The service shares the notice through the
coordination service, and reached_preemption_sync_point picks one step
that every process agrees on, so the checkpoint written there is
complete. A process outside any pool has no such service, so the notice
is SIGTERM itself, which this object catches until close. Only the main
thread can set a signal handler, so a notice created on another thread
outside a pool leaves SIGTERM’s default effect. A pool with the
preemption service turned off (jax_enable_preemption_service) gets no
notice, and SIGTERM ends it as usual.
PreemptionNotice.close
Section titled “PreemptionNotice.close”def close() -> NonePut back the SIGTERM handler this notice replaced.
PreemptionNotice.reached
Section titled “PreemptionNotice.reached”def reached() -> boolReturn whether to stop here, at this check.
In a pool, that is whether every process agreed to stop at this check, the sync point counting each check every process makes in step (each step’s, and each step of a cooldown branch), so it grows across a run however its steps run; a pool without the preemption service never stops. A process outside a pool stops once SIGTERM has arrived.
prepare_process
Section titled “prepare_process”def prepare_process( wandb: Wandb | None = None, multi_host: bool | None = None, xla_flags: str | None = None, compilation_cache_dir: str | None = None, cluster: ClusterEnvironment | None = None,) -> NoneSet the environment variables and XLA flags, raise the soft file descriptor limit, and join the JAX process pool.
wandb is the run’s dew.training.tracker.Wandb, or None for a run without a
tracker. Only its offline switch is read, and it has to be read before
wandb opens a run. multi_host=True requires a pool, False never joins
one, and None joins the pool the environment describes, if any.
cluster is the scheduler that started the run’s processes
(dew.clusters.ClusterEnvironment): its pool is joined, whatever
multi_host would detect, and given with multi_host=False it is refused,
as the two contradict each other. None reads the first of
dew.clusters.ENVIRONMENTS this process runs under, dew launch’s first.
compilation_cache_dir names the directory of JAX’s persistent
compilation cache, which a pool across hosts must keep on storage they
share; without one the process caches under the user’s cache directory
where its pool shares it (dew._cache.persist_compilations), and
JAX_ENABLE_COMPILATION_CACHE=false turns the cache off.
xla_flags reaches XLA through XLA_FLAGS, which XLA parses once per
runtime it loads, at the first use of its debug options. Flags set here
before the backend opens are written into XLA_FLAGS, as XLA may not have
parsed them yet; where import dew has loaded jaxlib, it may have, and
Dew warns that they are unproven (dew._devices.flags_hold). Setting
XLA_FLAGS before the process starts makes them certain, as Dew’s command
line does for a run’s trainer.xla_flags. Flags asked for after the
backend opened are refused.
A state kept on the CPU (Layout(host=("variables",))) needs one CPU
device per local accelerator, which the trainer checks on its own mesh
(dew.distributed.host.companion_mesh).
run_timestamp
Section titled “run_timestamp”def run_timestamp() -> strReturn process 0’s wall clock as %Y-%m-%d_%H:%M:%S, on every process.
The default run name includes it, and that name is the checkpoint directory every process writes into. So a process that read its own clock a second later would write into a different directory.
unprepared
Section titled “unprepared”def unprepared( multi_host: bool | None, xla_flags: str | None, compilation_cache_dir: str | None,) -> list[str]Which of prepare_process’s options a run states that this process does not carry: a flag of
xla_flags XLA_FLAGS did not carry while the backend was closed (flags_set), a compilation cache
JAX keeps elsewhere, a pool it has not joined (multi_host=True; False only says never to join one).
Each acts by the time JAX opens its backend, and prepare_process sets none after
(_set_environment), so a run trained in a process no prepare_process set up with them would run
without them.