Skip to content

dew.sampling

The reverse process for diffusion, and decoding for language models.

Each name is imported only when it is used (dew._exports), so a module that reads one of the package’s modules imports that module alone.

NameSummary
APGAdaptive projected guidance, combining raw outputs as Diffusers’ AdaptiveProjectedGuidance does. Documented in dew.sampling.guidance.
CFGInterval-limited classifier-free guidance. Documented in dew.sampling.guidance.
DDIMThe DDIM update, where eta sets the stochasticity: 0 is deterministic, 1 is DDPM-like. Documented in dew.sampling.solvers.
DDPMExact ancestral sampler for the reverse diffusion SDE. Documented in dew.sampling.solvers.
DEISDEIS (Zhang and Chen 2023, arXiv 2204.13902) in its log-rho multistep form, Diffusers 0.34.0’s DEISMultistepScheduler: the exponential integrator of eps with the polynomial-in-log(rho) interpolation of the last outputs, rho = sigma / alpha, integrated in closed form over the step. Documented in dew.sampling.solvers.
KDPM2k-diffusion’s DPM-Solver-2 (sample_dpm_2), the update of Diffusers 0.34.0’s KDPM2DiscreteScheduler, and with ancestral its sample_dpm_2_ancestral and KDPM2AncestralDiscreteScheduler. Documented in dew.sampling.solvers.
LMSLinear multistep over dx/dsigma = (x - x_0) / sigma, k-diffusion’s sample_lms and Diffusers 0.34.0’s LMSDiscreteScheduler: the last order derivatives interpolated by the Lagrange polynomial through their sigmas and integrated over the step, in closed form where Diffusers quadratures; the order grows with the history. Documented in dew.sampling.solvers.
PNDMPNDM as Diffusers 0.34.0’s PNDMScheduler: Adams-Bashforth over eps with DDIM as the transfer. Documented in dew.sampling.solvers.
RK4Classical fourth-order Runge-Kutta over dx/dsigma = eps on a variance-exploding schedule. Documented in dew.sampling.solvers.
TCDTrajectory consistency sampling, as Diffusers 0.34.0’s TCDScheduler. Documented in dew.sampling.solvers.
AutoguidanceGuides a model with a weaker version of itself (autoguidance). Documented in dew.sampling.guidance.
BeamRuns deterministic beam search over one shared prefill.
CFGPlusPlusGuides the clean prediction and renoises with the unconditional noise (CFG++). Documented in dew.sampling.guidance.
ConsistencyMultistep consistency sampling, the update of Diffusers 0.34.0’s LCMScheduler. Documented in dew.sampling.solvers.
DPMSolverMultistepDPM-Solver and DPM-Solver++ as multistep integrators in lambda = log(alpha) - log(sigma). Documented in dew.sampling.solvers.
DPMSolverSDEDiffusers 0.34.0’s DPMSolverSDEScheduler, k-diffusion’s sample_dpmpp_sde midpoint solver over a Brownian tree. Documented in dew.sampling.solvers.
DPMSolverSinglestepDiffusers 0.34.0’s grouped DPM-Solver updates from each group’s anchor. Documented in dew.sampling.solvers.
EulerThe DDIM update written as an Euler step of the probability flow ODE. Documented in dew.sampling.solvers.
EulerAncestralEuler with k-diffusion’s ancestral noise injection (get_ancestral_step, eta 1). Documented in dew.sampling.solvers.
FlowSDESamples a rectified-flow Process with Flow-GRPO’s Euler-Maruyama solver. Documented in dew.sampling.flow.
FlowTrajectoryA reverse trajectory, with batch-major states and joint log densities. Documented in dew.sampling.flow.
GaussianTransitionAn isotropic Gaussian transition with one variance per batch row. Documented in dew.sampling.flow.
GenerationThe prompt and padded continuation of each row, with response-aligned log-probabilities.
GrammarA token automaton on the device that keeps sampled text inside a regex or JSON schema.
GuidanceWhat sample guides with: a walk over the steps for one denoiser, and the modules that walk runs besides the denoiser’s model (modules, none for most). Documented in dew.sampling.guidance.
HeunHeun’s second-order method: an Euler step, the derivative re-evaluated at its end, and their average. Documented in dew.sampling.solvers.
LogitsTransformA pure rewrite of [rows, vocab] scores, applied before the draw. Documented in dew.sampling.decoding.
MultiStepDPMA third order multistep integrator of dx/dsigma = eps on a variance exploding schedule, from finite differences of the last three eps: FlaxDiff’s MultiStepDPM, which despite the name is not DPM-Solver. Documented in dew.sampling.solvers.
SampleDraws every row independently, one token per step.
SamplingCommon generation controls that decide how tokens are drawn and when a row stops.
SolverThe interface for one diffusion solver step and the state it keeps between steps. Documented in dew.sampling.solvers.
SpeculativeDrafts with the model’s prediction depths or block drafter, and verifies with the model itself.
StepStateWhat a transform or a criterion sees at one decode step. Documented in dew.sampling.decoding.
StoppingA pure per-row finish test over the tokens a step just drew. Documented in dew.sampling.decoding.
StrategyThe device loop that runs one generation request.
UniPCUniPC, a unified predictor and corrector in lambda, as Diffusers 0.34.0’s UniPCMultistepScheduler. Documented in dew.sampling.solvers.
generateGenerate continuations from model inputs, or from token ids for a text-only prompt.
sampleRun the reverse process from x_T with solver and return the final sample.
with_ids_ofReturn policy with the EOS and pad ids it leaves None taken from defaults.

dataclass source

class Beam(
width: int = struct.field(pytree_node=False, default=1),
length_penalty: float = struct.field(pytree_node=False, default=1.0),
early_stopping: bool | str = struct.field(pytree_node=False, default=False),
stop_ids: int = struct.field(pytree_node=False, default=1),
)

Runs deterministic beam search over one shared prefill.

The bookkeeping follows _beam_search in Transformers 5.16.1. Each step scores every live beam’s continuations and keeps the best keep of them, max(2, 1 + stop_ids) * width, so that width live beams always remain. stop_ids is the number of tokens that can end a beam. The kept continuations that a stopping criterion ended move into the completed set, with their score divided by their generated length raised to length_penalty, and the search continues with the rest.

early_stopping takes the reference’s three settings, which decide when a prompt is no longer worth extending from an estimate of the best score its live beams can still reach. False estimates it from the current length. True does the same and also stops recording completed beams once every slot of the completed set is full. “never” estimates it from the whole budget when the penalty rewards length.

The prompt is prefilled once and its cache row is copied into width rows. Every step gathers each beam’s parent row through DecodeOps.reindex, so a branched beam decodes exactly like a separately selected prefix. Parameters are never mapped.

n is how many completed beams to return, which is separate from the search width, and an n larger than width raises ValueError. A selected path is a search result and not a draw, so its behaviour log probability is zero. The raw log probabilities are still the model’s own for the tokens on the path.

keep: int

The number of continuations a step keeps, as beams_to_keep upstream.

dataclass source

class Generation(
rows: int | None = struct.field(pytree_node=False, default=None),
decoder: Callable[[ArrayLike, ArrayLike, int], tuple[str, ...]] | None = struct.field(pytree_node=False, default=None),
)

The prompt and padded continuation of each row, with response-aligned log-probabilities.

lengths counts each row’s response actions, including EOS. terminated is true where a stopping criterion (EOS by default) ended the row and false where the length limit did. Both log-probability arrays have shape [B, max_new_tokens], and only positions below lengths are valid. behavior_log_probs describes the distribution that actually drew each action, after the whole transform chain. raw_log_probs describes the unmodified model policy.

A request for n continuations per prompt gives every array [B * n, ...] rows: prompt zero’s n continuations, then prompt one’s. Each row has its own length, termination and likelihoods.

The arrays keep the placement the task ran with. On a mesh they are global arrays whose rows are split over the batch axes and padded to the device count. host() reads this process’s rows real rows back as host arrays, and text decodes them through the task’s processor.

text: tuple[str, ...]

Each real row’s valid continuation, decoded on first access (row_texts).

def host() -> Generation[np.ndarray]

Return this process’s real rows as host arrays, without the padding rows that fill the devices.

dataclass source

class Grammar()

A token automaton on the device that keeps sampled text inside a regex or JSON schema.

dew.sampling.guided.regex and dew.sampling.guided.json_schema compile one against a tokenizer’s vocabulary, and Sample(grammar) applies it. transitions[state, class] is the state a token of that class leads to, or -1 where the token is not allowed, and classes[token] is the token’s class. State 0 is the state before the first draw, so a zeroed carry starts a row.

def start(rows: int) -> jax.Array
def masked(state: jax.Array, logits: jax.Array) -> jax.Array

Return logits [rows, vocab] with the tokens each row’s state forbids set to -inf.

def guiding(
transform: Callable[[StepState, jax.Array], jax.Array],
state: jax.Array,
) -> Callable[[StepState, jax.Array], jax.Array]

Return transform, applied after masking the tokens each row’s state forbids. A LogitsChain stays one, the mask its first transform, so a terminal Greedy still draws by argmax (strategies.draw).

def advanced(state: jax.Array, token: jax.Array, drawn: jax.Array) -> jax.Array

Return each row’s state after token; a row that did not draw keeps its state.

A drawn token that the state forbids can only come from a transform that forces a token after the mask, such as ForcedEOS. Such a token fails the device check, so the text never leaves the language.

dataclass source

class Sample(grammar: Grammar | None = None)

Draws every row independently, one token per step.

This is the loop generate runs when a request names no strategy. A prompt’s continuations share its prefill (continuations).

grammar restricts every draw to a regex or JSON schema (dew.sampling.guided). Each row keeps its own automaton state, and the tokens the automaton forbids get the score -inf before the transform chain runs. So the chain filters and samples within the language, and the raw likelihood is still the model’s own.

dataclass source

class Sampling(
temperature: float = 1.0,
top_k: int | None = None,
eos_token_ids: int | tuple[int, ...] | None = None,
pad_token_id: int | None = None,
top_p: float = 1.0,
min_p: float = 0.0,
repetition_penalty: float = 1.0,
presence_penalty: float = 0.0,
frequency_penalty: float = 0.0,
no_repeat_ngram_size: int = 0,
min_new_tokens: int = 0,
typical_p: float = 1.0,
stop: tuple[str, ...] = (),
)

Common generation controls that decide how tokens are drawn and when a row stops.

Zero temperature is deterministic argmax, and top_k=None keeps the whole vocabulary. EOS counts as a sampled action; the output slots after it contain pad_token_id and have no likelihood.

The repetition penalty, no_repeat_ngram_size, min_new_tokens and typical_p are Transformers’ controls of the same names, and presence_penalty and frequency_penalty are vLLM’s. Each one’s default changes nothing. stop ends a row whose text ends with one of the strings; a task compiles them against its processor.

eos_token_ids and pad_token_id, Transformers’ names, belong to the model and its tokenizer, so where a policy leaves them None, a task fills in its own, as Transformers does. Plain generate has no task, so it stops on EOS only when the policy names one, and pads with 0 unless the policy names a pad_token_id. The block and masked decoders take the same two names.

A Sampling value compiles to its transforms in Transformers’ order when a request has no explicit logits chain. An explicit chain replaces those transforms. The EOS and stop-string criteria still join the request’s stopping criteria.

pad: int

The id in the output slots after EOS, which is pad_token_id, or 0 when it is None.

stops: tuple[int, ...]

The EOS ids that end a draw, or an empty tuple when the policy names no EOS.

def active(names: Sequence[str]) -> list[str]

Return the controls in names that this policy sets away from their defaults.

A client for a remote engine that cannot apply some controls the way Dew does passes their names here, and refuses a policy that sets any of them so the engine never draws without them.

def transforms() -> tuple[LogitsTransform, ...]

Return the full default chain that a request without explicit transforms runs.

At zero temperature the chain ends in the argmax and runs none of the sampling-only filters, as Transformers does with do_sample=False. To add a transform of your own, pass logits=policy.transforms() + (mine,).

def criteria() -> tuple[Stopping, ...]

Return the EOS criterion this policy adds after a caller’s criteria, empty when it names no EOS.

dataclass source

class Speculative(
block: int = struct.field(pytree_node=False, default=4),
confidence: float = struct.field(pytree_node=False, default=0.0),
)

Drafts with the model’s prediction depths or block drafter, and verifies with the model itself.

The acceptance rule is algorithm 1 of arXiv 2211.17192, as _speculative_sampling in Transformers 5.16.1 applies it. The first candidate of a block is an ordinary target draw, so it is always accepted. The model’s prediction depths chain the remaining candidates from the target’s last hidden state and each candidate’s embedding, as vLLM’s Qwen3_5MultiTokenPredictor does. A block drafter (DeepSeek-V4.1’s DSpark) instead drafts all the candidates after the first in one pass, drawing each from its position’s logits as the pass reaches it.

A proposed token x is accepted with probability min(1, p(x) / q(x)), where p is the target’s post-transform distribution and q is the draft’s actual one, compared as a log ratio. At the first rejection, the replacement is drawn from the normalized positive part of p - q, and a block with nothing rejected draws a bonus token from p. So the emitted tokens have exactly the distribution Sample would give them, token for token, though not the same draws at one seed.

Every emitted token, including a replacement or a bonus, records the target’s post-transform log probability as its behaviour log probability and the model’s own log probability as its raw value. The draft’s q, the acceptance probability and the residual are never recorded, because none of them is the distribution the emitted token came from.

Each iteration proposes block candidates (at least 2), so every collective keeps the same size. The target cache is saved before the block and the accepted prefix is replayed into it, because a recurrent mixer’s state is a running summary that no cursor can rewind. The prediction cache is rebuilt the same way. A block that continues emits two or more tokens unless the budget ends first, or one where a row walks across a cache rebuild (block_step), in at most block blocks, so ceil(budget / 2) iterations bound the loop, ceil((budget + block) / 2) where the model’s caches rebuild.

confidence stops the draft after the first candidate to which the draft itself gives a probability below confidence, as the reference’s ConfidenceCriteria does. The later candidates are still computed, at the same shapes, but cannot be accepted. Calling the strategy on a model that has neither prediction depths nor a block drafter raises ValueError.

class source

class Strategy(Protocol)

The device loop that runs one generation request.

A strategy is called with the prefilled DecoderState, the initial StepState, the DecodeOps that advance the model and gather its cache rows, the composed transform chain, the stopping criterion, the token budget and the number of continuations. It returns one Draws record per output row. Sample, Beam and Speculative use nothing else, so your own strategy is a callable with the same signature. Only Speculative drafts, so only it is given the model’s prediction depths and block drafter (DecodeOps.propose).

function source

def generate(
model: nn.Module,
params: Variables,
inputs: ModelInputs | ArrayLike | Sequence[Sequence[int]],
max_new_tokens: int,
*,
key: int | jax.Array,
sampling: Sampling = _DEFAULT_SAMPLING,
n: int = 1,
logits: Transforms | None = None,
stopping: Criteria | None = None,
strategy: Strategy | None = None,
trips: int | None = None,
prefix: int | Literal['prompt'] | None = None,
progress: Progress = END,
) -> Generation

Generate continuations from model inputs, or from token ids for a text-only prompt.

params is the full variables dict, {'params': ...}. ModelInputs.token_fields["attention_mask"] marks the real tokens; without a mask, every token is real. Every row must contain at least one real token. Prefill evaluates the conditioning once, and decoding reuses the cache and logical-position state the model keeps. Each cache compacts the real input tokens and leaves paused rows unchanged.

Parameters keep their placement, and decode at the model’s compute dtype: a call casts a parameter stored wider to it once, before its loop (dew.interop.weights.decoding_storage), and holds that copy beside the caller’s weights while it runs (fp32 weights under bf16 compute: half their bytes again). On a mesh, the rows are split over its batch axes and the result keeps that sharding; Generation.host() reads a process’s own rows back. All cooperating processes must use the same input shapes, effective decoding components, padding id and continuation count. The decode loops have fixed bounds, and the processes agree on any block they skip. Keys fold in the global row index and response position, so a pool draws what one process draws for the same rows.

The n continuations of each prompt share its prefill and come back as n consecutive rows of every array, in prompt order. Continuation zero of a prompt draws with that prompt’s own key, so n=1 and continuation zero of any larger request are the same draw.

logits is the whole transform chain, in the order it runs. None means the chain sampling compiles to, and () runs no transform. stopping adds criteria beside the policy’s EOS criterion and never replaces it. strategy replaces the per-row draw loop; None uses Sample. A policy with stop strings raises ValueError, because generate has no tokenizer to compile them against.

trips, at least max_new_tokens, scans that many draws, so calls of several budgets share one compiled loop (TextGeneration buckets them); the budget still bounds what the request draws as far as a transform or a search reads it (ForcedEOS, Beam), and the columns past it come back for the caller to cut.

progress is the run’s progress a model that reads one draws at (dew.nn.progress): its prefill’s and every step’s forward read it, an input of the compiled loop like the weights, so a run’s collection draws from its policy at the updates it committed; END, a finished run’s.

prefix serves a prefix run (LMObjective’s): each prompt that carries no blocks of its own reads its first prefix real tokens, or with "prompt" all of them, as one block (served_blocks). They are made of each process’s own rows inside the request’s agreed check, and a prompt with fewer real tokens than an int prefix is refused there, by every process of a pool together.

function source

def sample(
denoise: Denoiser | DiscreteDenoiser,
x_T: jax.Array,
steps: int | None = None,
*,
solver: Solver[StateT],
guidance: Guidance | None = None,
key: int | jax.Array | RowKeys,
times: ArrayLike | Sequence[float] | None = None,
initial: StateT | None = None,
final_denoise: bool = True,
record: Callable[[jax.Array, StateT, jax.Array], RecordT] | None = None,
) -> jax.Array | tuple[jax.Array, RecordT]

Run the reverse process from x_T with solver and return the final sample.

The time grid runs from T to 0. The solver takes one step across each interval, and the result is the model’s clean prediction at the last point. Pass exactly one of steps and times. With steps, the process supplies its grid (times(steps)): that many points for a Gaussian process, and MDLM’s grid of that many reveal steps for a masked one (DiscreteProcess.times). An explicit times grid is used as the trajectory, descending and concrete, for a source whose solver pairs its own sigma and model-time tables. Its length sets the number of steps, so a grid of steps + 1 points ending on the terminal is allowed, and a single point takes no solver step. final_denoise=False returns the last point’s state without the closing clean prediction, which is how those solvers end.

A caller that compiles one walk for many grids of a length passes times as data, traced, with initial, the solver’s state before the first step that it built on the host from the concrete grid (solver.init), whose checks and tables need the grid’s values.

record reads each step’s new state, the solver’s state after it and the clean prediction the step took, and the sample comes back with those readings stacked over the steps, (sample, records); Flow-GRPO records each transition and its density this way (FlowSDE.trajectory), and a figure of the walk each step’s clean prediction. A record needs at least one step.

denoise is process.denoiser(...), which holds the process the solver reads, and guidance wraps it. Every step’s noise comes from key folded with the step index, so a trajectory is reproducible from one key.

key may be RowKeys, one key a row, for a solver that draws row by row (RowSolver, Unmask): each row’s draws are its own key’s, folded with the step index, while the model still runs once over the batch. Another solver refuses them by name.

The trajectory is one lax.scan, traced at each call. Under a caller’s jax.jit it compiles once, which is how the pipelines and objectives call it. Called eagerly, it traces and compiles the scan again every time.

function source

def with_ids_of(policy: Sampling, defaults: Sampling) -> Sampling

Return policy with the EOS and pad ids it leaves None taken from defaults.

A task’s policy holds its model’s and tokenizer’s ids, so a call that replaces the policy keeps stopping and padding where the task does.