Skip to content

dew.objectives

NameSummary
AuxEverything a loss returns besides its statistics. Documented in dew.objectives.base.
DistillationObjectiveMixes the student’s own loss with a frozen teacher’s soft targets.
EMASpecWhich leaves the EMA copy tracks, and how fast it follows them. Documented in dew.objectives.base.
ObjectiveDefines what is learned: the parameters, the loss and what evaluation produces. Documented in dew.objectives.base.
PredictionWhat a token objective scored a batch with, for comparison with a teacher. Documented in dew.objectives.base.
RatioA loss numerator and its denominator, kept apart until the loss is reduced. Documented in dew.objectives.base.
ShownHow the training display shows one reported metric. Documented in dew.objectives.base.
StepWhat the trainer tells an objective about the current step. Documented in dew.objectives.base.
WeightedA loss of several terms, each a Ratio normalized by its own mass, summed with weights: sum_i weight_i * term_i.mean(), which has mass where any term of nonzero weight has.

class source

class DistillationObjective(
student: Objective[Loss, Effects],
teacher: Objective[Loss, Effects],
*,
alpha: Weight = 0.5,
temperature: Weight = 1.0,
features: Sequence[tuple[int, int]] = (),
beta: Weight = 0.0,
feature_loss: FeatureLoss = 'cosine',
divergence: Literal['full', 'selected'] = 'full',
tokenizers: tuple[str, str] | None = None,
)

Mixes the student’s own loss with a frozen teacher’s soft targets.

def data(tokenizer: Tokenizer | None = None) -> ObjectiveData

The student’s record and batch steps: the teacher scores the batch the student reads.

def real_lengths(batch: Batch) -> np.ndarray

The student’s real lengths of the batch it reads.

def scored_tokens(batch: Batch) -> np.ndarray

The targets the student scores of the batch it reads.

def row_tokens(batch: Batch) -> int | None

The tokens each row of the batch the student reads holds.

def inference_record() -> JSON

The student’s record: a saved distillation loads as the student it trained, on the student’s own tree, the teacher and the projections left unread (UNREAD).

def program_key() -> tuple[ProgramModule, ...]

The student’s modules, then the teacher’s, which the step does not train. The step reads both models’ whole logits (predict), so no head of either is fused.

def towers() -> tuple[Held, ...]

The student’s towers, then the teacher’s.

def substitute(modules: Sequence[nn.Module]) -> None
def tile_head(tile: tuple[int, int] | None = None) -> str | None

Tile the student’s head, the vocabulary head this objective trains.

def restore_head(tile: tuple[int, int] | None, *, fused: bool) -> None
def held_variables() -> Variables | None

Return the student’s held tree, if any, with the teacher’s under teacher.

def init(key: jax.Array, variables: Variables | None = None) -> Variables
def stage_sites(store: Variables, stages: int) -> tuple[DecoderBank, ...]

The student’s runs a pipeline over stages stacks (Objective.stage_sites), which its tree, the run’s without the teacher and the projections, holds where the run’s does: the teacher reads no update, whose layers stay whole.

def student_variables(params: Variables) -> Variables

Return the student’s own tree, without the teacher or the feature projections.

The student’s methods read this tree, and a distilled checkpoint passes it on to a plain student run.

def model_variables(variables: Variables) -> Variables

Return the student’s model tree, without the teacher or the feature projections.

def loss(variables: Variables, batch: Batch, step: Step) -> tuple[Ratio, Aux[Effects]]
def apply_effects(variables: Variables, effects: Effects) -> Variables
def projected(params: Variables, *, updated: bool) -> Variables

The student’s projection of its own parameters; the feature projections as they are.

def evaluate(params: Variables, batch: Batch, step: Step) -> Artifacts | None
def preview(
params: Variables,
batch: Batch,
step: Step,
*,
scored: Artifacts | None = None,
) -> Artifacts | None
def build_task(variables: Variables, *, processor: Processor | None | Omitted = OMITTED)

Return the student’s task, without the teacher.

dataclass source

class Weighted(weights: tuple[float, ...] = struct.field(pytree_node=False))

A loss of several terms, each a Ratio normalized by its own mass, summed with weights: sum_i weight_i * term_i.mean(), which has mass where any term of nonzero weight has.

Each term sums across microbatches and devices as a Ratio does, so every term is its mean over the whole update before the weights combine them (Objective.reduce_loss).

def mean() -> tuple[jax.Array, jax.Array]

Return the weighted sum of the terms’ means, and whether any term of nonzero weight has mass.