Skip to content

dew.training.cooldown

Cooldown branches of a run (Hägele et al. 2024, “Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations”, arXiv:2405.18392): at chosen points of a run on a constant or slowly decaying rate, a branch from the run’s state trains on with its rate decayed to zero, is scored by the run’s validation suites as cooldown/<split>/<metric>, and is discarded, the run continuing as though none ran. A warmup-stable-decay schedule (dew.optim.Linear’s decay_start) so reads what a run annealed at each point would score, without a run for each.

NameSummary
BranchOne cooldown point as a run resolved it: a step, or a committed-token count whose passing step branches (Counted).
CooldownsWhere a run branches into cooldowns, and how each decays.

dataclass source

class Branch(at: int | Counted, cooldowns: Cooldowns)

One cooldown point as a run resolved it: a step, or a committed-token count whose passing step branches (Counted).

def pending(current: int, accumulation: int) -> Pending

The branch from step current as it runs, resolved: its step, steps and decay, which a run cut short inside it keeps to run it again (Checkpoints.cooldown), whatever its config then.

def due(step: int, counts: Mapping[str, tuple[int, int]] | None = None) -> bool

Whether the step step, which brought each counted clock’s count from the second of its counts to the first, branches: at its step, or where it crosses its point.

def recorded() -> JSON

The point as a run’s start records it (dew.telemetry.FitStarted.cadences).

dataclass source

class Cooldowns(
at: tuple[int | str, ...],
length: int | str = '20%',
shape: Literal['1-sqrt', 'linear'] = '1-sqrt',
)

Where a run branches into cooldowns, and how each decays.

Each of at is a step, or a share of the run such as 20%, resolved as a cadence’s is (dew.training.budget.every_step): the whole updates nearest that share of its planned steps, or on a token budget alone the step whose commit passes that share of its tokens. A branch trains length more steps, or a share of the steps the run has taken, such as 20%, the whole updates nearest it: the run’s own batches from where it stands, at its own rate times a decay from 1 to 0 over the branch’s updates, 1-sqrt (1 - sqrt(u / U), the paper’s best) or linear (1 - u / U, WSD’s linear decay: Linear(decay_start=S, decay_steps=S + U) from a constant rate is the same rates).

def branches(
steps: int | None,
tokens: int | None,
accumulation: int,
examples: int | None = None,
) -> tuple[Branch, ...]

Each point resolved against a run of steps microsteps, a budget of tokens alone, or a bound of examples no pass counts steps ahead of, accumulation microsteps an update (every_step); a point of steps inside an update, whose branch would begin with that update’s window part filled, is refused.

def factor(update: int, updates: int) -> float

The share of the run’s rate a branch of updates updates trains its updateth at.

def steps(current: int, accumulation: int) -> int

The steps of a branch from step current: length, or its share of the steps so far, the whole updates nearest it and at least one. A length of steps no whole number of updates is refused, as its last microsteps would commit none.