dew.training.cooldown
Cooldown branches of a run (Hägele et al. 2024, “Scaling Laws and Compute-Optimal Training Beyond Fixed
Training Durations”, arXiv:2405.18392): at chosen points of a run on a constant or slowly decaying rate,
a branch from the run’s state trains on with its rate decayed to zero, is scored by the run’s validation
suites as cooldown/<split>/<metric>, and is discarded, the run continuing as though none ran. A
warmup-stable-decay schedule (dew.optim.Linear’s decay_start) so reads what a run annealed at each
point would score, without a run for each.
| Name | Summary |
|---|---|
Branch | One cooldown point as a run resolved it: a step, or a committed-token count whose passing step branches (Counted). |
Cooldowns | Where a run branches into cooldowns, and how each decays. |
Branch
Section titled “Branch”class Branch(at: int | Counted, cooldowns: Cooldowns)One cooldown point as a run resolved it: a step, or a committed-token count whose passing step
branches (Counted).
Branch.pending
Section titled “Branch.pending”def pending(current: int, accumulation: int) -> PendingThe branch from step current as it runs, resolved: its step, steps and decay, which a run
cut short inside it keeps to run it again (Checkpoints.cooldown), whatever its config then.
Branch.due
Section titled “Branch.due”def due(step: int, counts: Mapping[str, tuple[int, int]] | None = None) -> boolWhether the step step, which brought each counted clock’s count from the second of its
counts to the first, branches: at its step, or where it crosses its point.
Branch.recorded
Section titled “Branch.recorded”def recorded() -> JSONThe point as a run’s start records it (dew.telemetry.FitStarted.cadences).
Cooldowns
Section titled “Cooldowns”class Cooldowns( at: tuple[int | str, ...], length: int | str = '20%', shape: Literal['1-sqrt', 'linear'] = '1-sqrt',)Where a run branches into cooldowns, and how each decays.
Each of at is a step, or a share of the run such as 20%, resolved as a cadence’s is
(dew.training.budget.every_step): the whole updates nearest that share of its planned steps, or on
a token budget alone the step whose commit passes that share of its tokens. A branch trains length
more steps, or a share of the steps the run has taken, such as 20%, the whole updates nearest it:
the run’s own batches from where it stands, at its own rate times a decay from 1 to 0 over the
branch’s updates, 1-sqrt (1 - sqrt(u / U), the paper’s best) or linear (1 - u / U, WSD’s linear
decay: Linear(decay_start=S, decay_steps=S + U) from a constant rate is the same rates).
Cooldowns.branches
Section titled “Cooldowns.branches”def branches( steps: int | None, tokens: int | None, accumulation: int, examples: int | None = None,) -> tuple[Branch, ...]Each point resolved against a run of steps microsteps, a budget of tokens alone, or a bound
of examples no pass counts steps ahead of, accumulation microsteps an update (every_step); a
point of steps inside an update, whose branch would begin with that update’s window part filled,
is refused.
Cooldowns.factor
Section titled “Cooldowns.factor”def factor(update: int, updates: int) -> floatThe share of the run’s rate a branch of updates updates trains its updateth at.
Cooldowns.steps
Section titled “Cooldowns.steps”def steps(current: int, accumulation: int) -> intThe steps of a branch from step current: length, or its share of the steps so far, the
whole updates nearest it and at least one. A length of steps no whole number of updates is
refused, as its last microsteps would commit none.