dew.eval.harness

Run a saved run as an lm-evaluation-harness model.

DewLM puts a TextGeneration behind lm-eval-harness’s TemplateLM, so any task suite runs against a run directory. The trainer’s own perplexity says how well a run predicts its training data and nothing about what it can do, which is the other question a suite answers.

lm_eval is an optional extra (pip install dew-ml[eval-harness]), so this module is the only one that imports it and dew.eval does not import this module: a caller who never asks for a harness never needs it installed. Importing this module registers the adapter under dew, which is what the harness’s registry reads, and lm_eval 0.4 has no plugin discovery of its own, so the import has to happen in the process that runs the command:

python -m dew.eval --model dew --model_args run=runs/shakespeare \
    --tasks hellaswag --limit 4

is lm_eval’s own command line with this module imported first. In a program that already imported it, plain lm_eval --model dew finds it too.

Everything that decides which tokens are scored is lm-eval’s own code: TemplateLM.loglikelihood splits each pair (moving a context’s trailing whitespace into the continuation, conditioning an empty context on the prefix token), and get_rolling_token_windows with make_disjoint_window cuts a long string so every token is scored exactly once. What this module adds is _loglikelihood_tokens, the row HFLM builds from each (context, continuation) pair, scored by the model’s own forward under jax.jit: slot i of the logits predicts token i + 1 of the row, so a continuation of n tokens is read at the n slots ending one before the row’s last token.

Name	Summary
DEFAULT_CONTEXT	The scoring window for a model that declares no max_seq_len.
TokenRequest	One _loglikelihood_tokens request: the strings, context ids, continuation ids.
DewLM	Puts a TextGeneration behind lm-eval-harness’s TemplateLM interface.

DEFAULT_CONTEXT

attribute source

DEFAULT_CONTEXT = 2048

The scoring window for a model that declares no max_seq_len.

TokenRequest

attribute source

TokenRequest = tuple[tuple[str, str] | None, list[int], list[int]]

One _loglikelihood_tokens request: the strings, context ids, continuation ids.

DewLM

class source

class DewLM(task: TextGeneration, *, batch_size: int = 1)

Puts a TextGeneration behind lm-eval-harness’s TemplateLM interface.

task is the run’s own generation task, with its model, its weights and its processor; batch_size is how many rows one scoring call runs at once. Requests keep their order, which is what the harness pairs its documents back up by.

Likelihoods are exact: the model’s log-softmax at the continuation’s own targets, summed. Generation is greedy unless a request’s gen_kwargs ask for a temperature, which is the harness’s own default and what a suite’s reported numbers assume.

eot_token_id: int

Return the id a row with no context is conditioned on: the policy’s EOS.

Sampling normalises its own field to a tuple, and declares the form a caller may write, so both spellings are read here.

prefix_token_id: int

Return the id a first token is conditioned on: the tokenizer’s BOS, else EOS.

This is HFLM.prefix_token_id. A vocabulary that starts every sequence with BOS scores its first token after BOS, and conditioning it on EOS instead would move every rolling and empty-context score.

max_length: int

Return how many ids one scoring row may hold, as the model declares it.

_ceiling is the same read a call already makes to size its cache, so a harness row and a generated row are bounded by the same field.

DewLM.from_run

def from_run(
    run: str,
    *,
    batch_size: int = 1,
    ema: bool = True,
    step: int | None = None,
    dtype: str | None = None,
) -> DewLM

Load the run in run as a harness model, the way dew.pipeline builds it.

DewLM.create_from_arg_string

def create_from_arg_string(
    arg_string: str,
    additional_config: dict | None = None,
) -> DewLM

Build the model from --model_args run=<directory>,batch_size=4.

DewLM.create_from_arg_obj

def create_from_arg_obj(arg_dict: dict, additional_config: dict | None = None) -> DewLM

Build the model from arguments already parsed, the route the CLI takes.

DewLM.tok_encode

def tok_encode(
    string: str,
    add_special_tokens: bool | None = None,
    **kwargs: int | None = {},
) -> list[int]

Encode string with the run’s own tokenizer, one row of ids.

The run’s processor decides special tokens the way it did in training, so add_special_tokens and the harness’s other integer options (left_truncate_len) are accepted and not read.

DewLM.tok_decode

def tok_decode(tokens: Sequence[int]) -> str

DewLM.loglikelihood_rolling

def loglikelihood_rolling(
    requests: list[Instance],
    disable_tqdm: bool = False,
) -> list[float]

Return each string’s own log-probability, every token scored once.

The windows are lm-eval’s: the first conditioned on the prefix token, each later one on the max_length ids before it, none overlapping in what it scores.

DewLM.generate_until

def generate_until(requests: list[Instance], disable_tqdm: bool = False) -> list[str]

Continue each context until one of its stop strings or its budget.

The stop strings cut the decoded text, so a sequence that spans two tokens ends the answer the way the harness expects it to.