Skip to content

dew.sampling.decoding

Logit transforms and stopping criteria for the decode loop.

Three concepts extend decoding. A LogitsTransform is a pure callable from the step state and [rows, vocab] logits to new logits. A Stopping is a pure callable from the step state and the tokens just drawn to a per-row finished flag; criteria combine with OR after every committed token. A Strategy in dew.sampling.strategies runs the device loop.

Transforms and criteria read StepState, which holds the token history and nothing about the model: no parameters and no cache. Every built-in here is a pytree, so a configuration holding arrays crosses jax.jit as data instead of entering a compilation cache key, and so do its numbers where they fix no shape (as_pytree): another temperature, penalty or length runs the same program. A plain function works as well, and jax.tree_util.Partial can hold array configuration for one.

The numerical reference is Transformers 5.16.1 generation/logits_process.py and generation/stopping_criteria.py, with two differences that follow from Dew’s decode loop. Each row reads its own unpadded history rather than the batch’s padded width, and frequency and presence penalties follow vLLM’s formula (model_executor/layers/utils.py), which Transformers does not implement.

NameSummary
BeginSuppressTokensRemoves tokens at one generated position, as SuppressTokensAtBeginLogitsProcessor.
EndOfSequenceFinishes a row that drew one of the EOS ids, as EosTokenCriteria.
EpsilonCutoffRemoves tokens below an absolute probability, as EpsilonLogitsWarper.
EtaCutoffEntropy-scaled cutoff, as EtaLogitsWarper, which takes the square root of its float32 epsilon on the device.
ExponentialDecayLengthPenaltyRaises the EOS score after start drawn tokens, as ExponentialDecayLengthPenalty.
FinishThe rows a step’s criteria finished, and those of them a criterion ended, as a stop does, rather than one cut short at a length (Stopping’s truncates): the terminated rows.
FinishingA request’s criteria as pytrees (components), those that end a row apart from those that cut it short at a length, which is told from each criterion as given (Truncates) before it is one: a built-in’s numbers ride as data (as_pytree).
ForcedBOSForces one token as the first of the whole sequence, as ForcedBOSTokenLogitsProcessor.
ForcedEOSForces EOS one step before the end, as ForcedEOSTokenLogitsProcessor.
FrequencyPenaltySubtracts penalty times each token’s count among the drawn tokens.
GreedyThe argmax as a distribution: zero on the best token, -inf elsewhere.
LogitsChainThe ordered transforms, keeping a terminal Greedy for the sampler.
LogitsTransformA pure rewrite of [rows, vocab] scores, applied before the draw.
MaxLengthFinishes a row once prompt and generated tokens reach length, as MaxLengthCriteria, which truncates it.
MaxNewTokensFinishes a row once it has drawn count tokens, which truncates it.
MinLengthSuppresses EOS until the whole sequence reaches length, as MinLengthLogitsProcessor.
MinNewTokensSuppresses EOS until count tokens are drawn, as MinNewTokensLengthLogitsProcessor.
MinPRelative filtering at p times the top probability, as MinPLogitsWarper.
NoRepeatNGramBans tokens that would repeat an n-gram of the row’s own history.
PresencePenaltySubtracts penalty from every token already drawn.
PromptNoRepeatNGramBans tokens that would repeat an n-gram of the prompt.
PromptRepetitionPenaltyRaises the scores of prompt tokens, as EncoderRepetitionPenaltyLogitsProcessor.
RemoveInvalidValuesMaps NaN to zero and infinities to the float range, as InfNanRemoveLogitsProcessor.
RenormalizeReplaces scores by their log softmax, as LogitNormalization.
RepetitionPenaltyDivides the positive scores of seen tokens by the penalty and multiplies negative ones.
SequenceBiasAdds a bias to the token that would complete each biased sequence.
StepStateWhat a transform or a criterion sees at one decode step.
StopStringsFinishes a row whose text ends with one of the compiled stop strings.
StoppingA pure per-row finish test over the tokens a step just drew.
SuppressTokensRemoves a fixed set of tokens, as SuppressTokensLogitsProcessor.
TemperatureDivides the logits by value, as TemperatureLogitsWarper.
TopHEntropy-budget filtering, as TopHLogitsWarper.
TopKKeeps the k highest scores, as TopKLogitsWarper.
TopPNucleus filtering, as TopPLogitsWarper.
TruncatesMarks a stopping criterion that cuts a row short at a length rather than ending it.
TypicalLocally typical filtering, as TypicalLogitsWarper.

dataclass source

class BeginSuppressTokens(
forced_bos: bool = struct.field(pytree_node=False, default=False),
)

Removes tokens at one generated position, as SuppressTokensAtBeginLogitsProcessor.

The reference suppresses the first drawn token, or the one after it where forced_bos (ForcedBOS) forces the first: transformers moves the begin index past the forced token only for a one-token prompt (generation/utils.py:1278-1282, 5.16.1), the rows where ForcedBOS fires. Each row decides by its own prompt, so rows of mixed prompt lengths suppress each at its own position.

def numerical() -> jax.tree_util.Partial
def apply(
tokens: jax.Array,
forced_bos: bool,
state: StepState,
logits: jax.Array,
) -> jax.Array

dataclass source

class EndOfSequence()

Finishes a row that drew one of the EOS ids, as EosTokenCriteria.

dataclass source

class EpsilonCutoff(epsilon: float = struct.field(pytree_node=False, default=0.0))

Removes tokens below an absolute probability, as EpsilonLogitsWarper.

def numerical() -> jax.tree_util.Partial
def apply(epsilon: float, state: StepState, logits: jax.Array) -> jax.Array

dataclass source

class EtaCutoff(epsilon: float = struct.field(pytree_node=False, default=0.0))

Entropy-scaled cutoff, as EtaLogitsWarper, which takes the square root of its float32 epsilon on the device.

def numerical() -> jax.tree_util.Partial
def apply(epsilon: float, state: StepState, logits: jax.Array) -> jax.Array

dataclass source

class ExponentialDecayLengthPenalty(
start: int = struct.field(pytree_node=False, default=0),
factor: float = struct.field(pytree_node=False, default=1.0),
eos: jax.Array = struct.field(default_factory=lambda: jnp.zeros((0,), jnp.int32)),
)

Raises the EOS score after start drawn tokens, as ExponentialDecayLengthPenalty.

The reference measures from start_index + prompt_width, which is the generated count used here, and adds |score| * (factor ** index - 1) so a negative score also rises. A removed EOS (-inf, a grammar’s mask) stays removed.

def numerical() -> jax.tree_util.Partial
def apply(
start: int,
factor: float,
eos: jax.Array,
state: StepState,
logits: jax.Array,
) -> jax.Array

class source

class Finish(NamedTuple)

The rows a step’s criteria finished, and those of them a criterion ended, as a stop does, rather than one cut short at a length (Stopping’s truncates): the terminated rows.

class source

class Finishing(NamedTuple)

A request’s criteria as pytrees (components), those that end a row apart from those that cut it short at a length, which is told from each criterion as given (Truncates) before it is one: a built-in’s numbers ride as data (as_pytree).

def of(criteria: Sequence[Stopping]) -> Finishing

dataclass source

class ForcedBOS(token: int = struct.field(pytree_node=False, default=0))

Forces one token as the first of the whole sequence, as ForcedBOSTokenLogitsProcessor.

def numerical() -> jax.tree_util.Partial
def apply(token: int, state: StepState, logits: jax.Array) -> jax.Array

dataclass source

class ForcedEOS(
eos: jax.Array = struct.field(default_factory=lambda: jnp.zeros((0,), jnp.int32)),
max_length: int | None = struct.field(pytree_node=False, default=None),
)

Forces EOS one step before the end, as ForcedEOSTokenLogitsProcessor.

eos may name several ids, all of which the forced step allows, as the reference allows every id its tensor holds. max_length counts prompt and generated tokens together. Left as None, the end is the request’s own token budget (StepState.budgets), so a caller that changes the budget per call forces EOS at the new end rather than at a length the task was built with, and a loop that scans past the budget forces it inside it.

def numerical() -> jax.tree_util.Partial
def apply(
eos: jax.Array,
max_length: int | None,
state: StepState,
logits: jax.Array,
) -> jax.Array

dataclass source

class FrequencyPenalty(penalty: float = struct.field(pytree_node=False, default=0.0))

Subtracts penalty times each token’s count among the drawn tokens.

This is vLLM’s formula, logits -= frequency_penalties * output_bin_counts (model_executor/layers/utils.py), which is also OpenAI’s frequency_penalty. It counts generated tokens only, not the prompt.

def numerical() -> jax.tree_util.Partial
def apply(penalty: float, state: StepState, logits: jax.Array) -> jax.Array

dataclass source

class Greedy()

The argmax as a distribution: zero on the best token, -inf elsewhere.

Sampling(temperature=0) compiles to this, so the draw stays the deterministic argmax with a behaviour log probability of exactly zero; a terminal Greedy lets the sampler take the argmax without building the point mass. Transforms before it still shape the argmax, as in greedy search. A row without a distribution (all removed, or NaN or +inf) passes through for the draw to refuse, rather than becoming a confident token.

dataclass source

class LogitsChain(transforms: tuple[LogitsTransform, ...])

The ordered transforms, keeping a terminal Greedy for the sampler.

An arbitrary transform after Greedy may restore a nondegenerate distribution, so only the final built-in Greedy proves an argmax draw. Like a function closure, the callable is hashed by identity, not by its captured processor arrays, when passed directly to jax.jit.

class source

class LogitsTransform(Protocol)

A pure rewrite of [rows, vocab] scores, applied before the draw.

dataclass source

class MaxLength(length: int = struct.field(pytree_node=False, default=0))

Finishes a row once prompt and generated tokens reach length, as MaxLengthCriteria, which truncates it.

def numerical() -> jax.tree_util.Partial
def apply(length: int, state: StepState, tokens: jax.Array) -> jax.Array

dataclass source

class MaxNewTokens(count: int = struct.field(pytree_node=False, default=0))

Finishes a row once it has drawn count tokens, which truncates it.

def numerical() -> jax.tree_util.Partial
def apply(count: int, state: StepState, tokens: jax.Array) -> jax.Array

dataclass source

class MinLength(
length: int = struct.field(pytree_node=False, default=0),
eos: jax.Array = struct.field(default_factory=lambda: jnp.zeros((0,), jnp.int32)),
)

Suppresses EOS until the whole sequence reaches length, as MinLengthLogitsProcessor.

def numerical() -> jax.tree_util.Partial
def apply(length: int, eos: jax.Array, state: StepState, logits: jax.Array) -> jax.Array

dataclass source

class MinNewTokens(
count: int = struct.field(pytree_node=False, default=0),
eos: jax.Array = struct.field(default_factory=lambda: jnp.zeros((0,), jnp.int32)),
)

Suppresses EOS until count tokens are drawn, as MinNewTokensLengthLogitsProcessor.

def numerical() -> jax.tree_util.Partial
def apply(count: int, eos: jax.Array, state: StepState, logits: jax.Array) -> jax.Array

dataclass source

class MinP(p: float = struct.field(pytree_node=False, default=0.0))

Relative filtering at p times the top probability, as MinPLogitsWarper.

def numerical() -> jax.tree_util.Partial
def apply(p: float, state: StepState, logits: jax.Array) -> jax.Array

dataclass source

class NoRepeatNGram(size: int = struct.field(pytree_node=False, default=0))

Bans tokens that would repeat an n-gram of the row’s own history.

This is the tensorised form of NoRepeatNGramLogitsProcessor: the current suffix is matched against every window, and a matching window bans the token that followed it. The window starting at the suffix itself needs one token more than the row has, so a suffix never bans its own successor.

dataclass source

class PresencePenalty(penalty: float = struct.field(pytree_node=False, default=0.0))

Subtracts penalty from every token already drawn.

This is vLLM’s logits -= presence_penalties * output_mask, which is also OpenAI’s presence_penalty. It reads generated tokens only, not the prompt.

def numerical() -> jax.tree_util.Partial
def apply(penalty: float, state: StepState, logits: jax.Array) -> jax.Array

dataclass source

class PromptNoRepeatNGram(size: int = struct.field(pytree_node=False, default=0))

Bans tokens that would repeat an n-gram of the prompt.

EncoderNoRepeatNGramLogitsProcessor builds its table from the encoder input and matches it against the decoder’s suffix. The prompt is the encoder input of a decoder-only model.

dataclass source

class PromptRepetitionPenalty(
penalty: float = struct.field(pytree_node=False, default=1.0),
)

Raises the scores of prompt tokens, as EncoderRepetitionPenaltyLogitsProcessor.

The reference inverts its argument, so a penalty above one rewards repeating the prompt. A decoder-only prompt is the encoder input here.

def numerical() -> jax.tree_util.Partial
def apply(inverted: float, state: StepState, logits: jax.Array) -> jax.Array

dataclass source

class RemoveInvalidValues()

Maps NaN to zero and infinities to the float range, as InfNanRemoveLogitsProcessor.

Nothing else in the chain repairs a broken distribution; an undefined draw raises instead. Add this transform if you want one sanitized.

dataclass source

class Renormalize()

Replaces scores by their log softmax, as LogitNormalization.

dataclass source

class RepetitionPenalty(penalty: float = struct.field(pytree_node=False, default=1.0))

Divides the positive scores of seen tokens by the penalty and multiplies negative ones.

The history is the row’s valid prompt and drawn tokens, as RepetitionPenaltyLogitsProcessor reads the whole input_ids.

def numerical() -> jax.tree_util.Partial
def apply(penalty: float, state: StepState, logits: jax.Array) -> jax.Array

dataclass source

class SequenceBias()

Adds a bias to the token that would complete each biased sequence.

This is SequenceBiasLogitsProcessor as a table: sequences is [count, width] right-aligned token ids, lengths their real lengths and bias the value added to the last id when the row’s suffix matches the preceding ones. A sequence longer than the row’s history is skipped, as the reference skips one longer than the context.

dataclass source

class StepState(
prompt_width: int = struct.field(pytree_node=False, default=0),
budget: jax.Array | None = None,
)

What a transform or a criterion sees at one decode step.

tokens is the fixed-capacity buffer of the prompt followed by the draw slots, [rows, prompt_width + budget], and valid marks the slots that hold a real token. A row’s history is therefore its own, whatever padding the prompt batch needed. step counts the tokens the row has committed, active marks the rows still generating, and keys holds one PRNG key per row. budget is the tokens each row’s request may draw, [rows], which can be fewer than the draw slots: a task scans a bucket of trips past it so budgets share one compiled loop, and a server’s slots hold its whole capacity. None leaves it the draw slots.

width: int

Slots in the buffer, prompt plus budget.

def budgets() -> jax.Array

Return the tokens each row’s request may draw (budget), [rows].

def total() -> jax.Array

Return the number of real tokens each row holds, prompt and generated together.

def history() -> tuple[jax.Array, jax.Array]

Return each row’s real tokens left-aligned, and how many there are.

Prompts pad wherever their batch needed it, so a transform that reads order (n-grams, biased sequences, stop strings) needs the row’s own tokens without holes. Padding slots hold -1, which no token id equals.

def prompt_history() -> tuple[jax.Array, jax.Array]

Return the prompt region’s real tokens left-aligned, and how many there are.

def commit(tokens: jax.Array, drawn: jax.Array) -> StepState

Return the state after the drawn rows append tokens at their next slot.

dataclass source

class StopStrings(
positions: int = struct.field(pytree_node=False, default=1),
ends: int = struct.field(pytree_node=False, default=1),
span: int = struct.field(pytree_node=False, default=1),
)

Finishes a row whose text ends with one of the compiled stop strings.

The tables come from stop_strings, which reads the tokenizer once. The device check is StopStringCriteria’s: it goes through the row’s tokens backwards, requires the last token to overlap the end of a stop string, and keeps matching earlier tokens against the positions they can occupy. A match counts only when the string touches the final token, so a string produced earlier does not stop the row later.

class source

class Stopping(Protocol)

A pure per-row finish test over the tokens a step just drew.

A criterion that cuts a row short at a length rather than ending it, as MaxNewTokens and MaxLength do, is a Truncates: a row it finishes is not terminated (Finish).

dataclass source

class SuppressTokens()

Removes a fixed set of tokens, as SuppressTokensLogitsProcessor.

dataclass source

class Temperature(value: float = struct.field(pytree_node=False, default=1.0))

Divides the logits by value, as TemperatureLogitsWarper.

def numerical() -> jax.tree_util.Partial
def apply(value: float, state: StepState, logits: jax.Array) -> jax.Array

dataclass source

class TopH(fraction: float = struct.field(pytree_node=False, default=1.0))

Entropy-budget filtering, as TopHLogitsWarper.

Tokens enter in probability order while the cumulative entropy of the truncated head stays within fraction of its total entropy, and the best token always enters, out of the reference’s fixed head of 100.

The two entropies are computed the way the reference computes them, and they are not the same expression. The budget is torch.distributions.Categorical.entropy, which clamps the log probabilities to the dtype’s minimum so a removed token contributes nothing. The running sum is the reference’s own -p * log(p), whose removed tokens are NaN, and a NaN ends the selection because every comparison against it is false. Substituting one for the other keeps a token the reference drops.

def numerical() -> jax.tree_util.Partial
def apply(fraction: float, state: StepState, logits: jax.Array) -> jax.Array

dataclass source

class TopK(k: int = struct.field(pytree_node=False, default=1))

Keeps the k highest scores, as TopKLogitsWarper.

dataclass source

class TopP(p: float = struct.field(pytree_node=False, default=1.0))

Nucleus filtering, as TopPLogitsWarper.

The ascending tail holding cumulative mass at most 1 - p is removed and the best token always survives.

def numerical() -> jax.tree_util.Partial
def apply(excluded_mass: float, state: StepState, logits: jax.Array) -> jax.Array

class source

class Truncates

Marks a stopping criterion that cuts a row short at a length rather than ending it.

dataclass source

class Typical(mass: float = struct.field(pytree_node=False, default=1.0))

Locally typical filtering, as TypicalLogitsWarper.

def numerical() -> jax.tree_util.Partial
def apply(mass: float, state: StepState, logits: jax.Array) -> jax.Array