dew.sampling
The reverse process for diffusion, and decoding for language models.
Each name is imported only when it is used (dew._exports), so a module that reads one of the
package’s modules imports that module alone.
| Name | Summary |
|---|---|
APG | Adaptive projected guidance, combining raw outputs as Diffusers’ AdaptiveProjectedGuidance does. Documented in dew.sampling.guidance. |
CFG | Interval-limited classifier-free guidance. Documented in dew.sampling.guidance. |
DDIM | The DDIM update, where eta sets the stochasticity: 0 is deterministic, 1 is DDPM-like. Documented in dew.sampling.solvers. |
DDPM | Exact ancestral sampler for the reverse diffusion SDE. Documented in dew.sampling.solvers. |
DEIS | DEIS (Zhang and Chen 2023, arXiv 2204.13902) in its log-rho multistep form, Diffusers 0.34.0’s DEISMultistepScheduler: the exponential integrator of eps with the polynomial-in-log(rho) interpolation of the last outputs, rho = sigma / alpha, integrated in closed form over the step. Documented in dew.sampling.solvers. |
KDPM2 | k-diffusion’s DPM-Solver-2 (sample_dpm_2), the update of Diffusers 0.34.0’s KDPM2DiscreteScheduler, and with ancestral its sample_dpm_2_ancestral and KDPM2AncestralDiscreteScheduler. Documented in dew.sampling.solvers. |
LMS | Linear multistep over dx/dsigma = (x - x_0) / sigma, k-diffusion’s sample_lms and Diffusers 0.34.0’s LMSDiscreteScheduler: the last order derivatives interpolated by the Lagrange polynomial through their sigmas and integrated over the step, in closed form where Diffusers quadratures; the order grows with the history. Documented in dew.sampling.solvers. |
PNDM | PNDM as Diffusers 0.34.0’s PNDMScheduler: Adams-Bashforth over eps with DDIM as the transfer. Documented in dew.sampling.solvers. |
RK4 | Classical fourth-order Runge-Kutta over dx/dsigma = eps on a variance-exploding schedule. Documented in dew.sampling.solvers. |
TCD | Trajectory consistency sampling, as Diffusers 0.34.0’s TCDScheduler. Documented in dew.sampling.solvers. |
Autoguidance | Guides a model with a weaker version of itself (autoguidance). Documented in dew.sampling.guidance. |
Beam | Runs deterministic beam search over one shared prefill. |
CFGPlusPlus | Guides the clean prediction and renoises with the unconditional noise (CFG++). Documented in dew.sampling.guidance. |
Consistency | Multistep consistency sampling, the update of Diffusers 0.34.0’s LCMScheduler. Documented in dew.sampling.solvers. |
DPMSolverMultistep | DPM-Solver and DPM-Solver++ as multistep integrators in lambda = log(alpha) - log(sigma). Documented in dew.sampling.solvers. |
DPMSolverSDE | Diffusers 0.34.0’s DPMSolverSDEScheduler, k-diffusion’s sample_dpmpp_sde midpoint solver over a Brownian tree. Documented in dew.sampling.solvers. |
DPMSolverSinglestep | Diffusers 0.34.0’s grouped DPM-Solver updates from each group’s anchor. Documented in dew.sampling.solvers. |
Euler | The DDIM update written as an Euler step of the probability flow ODE. Documented in dew.sampling.solvers. |
EulerAncestral | Euler with k-diffusion’s ancestral noise injection (get_ancestral_step, eta 1). Documented in dew.sampling.solvers. |
FlowSDE | Samples a rectified-flow Process with Flow-GRPO’s Euler-Maruyama solver. Documented in dew.sampling.flow. |
FlowTrajectory | A reverse trajectory, with batch-major states and joint log densities. Documented in dew.sampling.flow. |
GaussianTransition | An isotropic Gaussian transition with one variance per batch row. Documented in dew.sampling.flow. |
Generation | The prompt and padded continuation of each row, with response-aligned log-probabilities. |
Grammar | A token automaton on the device that keeps sampled text inside a regex or JSON schema. |
Guidance | What sample guides with: a walk over the steps for one denoiser, and the modules that walk runs besides the denoiser’s model (modules, none for most). Documented in dew.sampling.guidance. |
Heun | Heun’s second-order method: an Euler step, the derivative re-evaluated at its end, and their average. Documented in dew.sampling.solvers. |
LogitsTransform | A pure rewrite of [rows, vocab] scores, applied before the draw. Documented in dew.sampling.decoding. |
MultiStepDPM | A third order multistep integrator of dx/dsigma = eps on a variance exploding schedule, from finite differences of the last three eps: FlaxDiff’s MultiStepDPM, which despite the name is not DPM-Solver. Documented in dew.sampling.solvers. |
Sample | Draws every row independently, one token per step. |
Sampling | Common generation controls that decide how tokens are drawn and when a row stops. |
Solver | The interface for one diffusion solver step and the state it keeps between steps. Documented in dew.sampling.solvers. |
Speculative | Drafts with the model’s prediction depths or block drafter, and verifies with the model itself. |
StepState | What a transform or a criterion sees at one decode step. Documented in dew.sampling.decoding. |
Stopping | A pure per-row finish test over the tokens a step just drew. Documented in dew.sampling.decoding. |
Strategy | The device loop that runs one generation request. |
UniPC | UniPC, a unified predictor and corrector in lambda, as Diffusers 0.34.0’s UniPCMultistepScheduler. Documented in dew.sampling.solvers. |
generate | Generate continuations from model inputs, or from token ids for a text-only prompt. |
sample | Run the reverse process from x_T with solver and return the final sample. |
with_ids_of | Return policy with the EOS and pad ids it leaves None taken from defaults. |
class Beam( width: int = struct.field(pytree_node=False, default=1), length_penalty: float = struct.field(pytree_node=False, default=1.0), early_stopping: bool | str = struct.field(pytree_node=False, default=False), stop_ids: int = struct.field(pytree_node=False, default=1),)Runs deterministic beam search over one shared prefill.
The bookkeeping follows _beam_search in Transformers 5.16.1. Each step
scores every live beam’s continuations and keeps the best keep of
them, max(2, 1 + stop_ids) * width, so that width live beams always
remain. stop_ids is the number of tokens that can end a beam. The kept
continuations that a stopping criterion ended move into the completed
set, with their score divided by their generated length raised to
length_penalty, and the search continues with the rest.
early_stopping takes the reference’s three settings, which decide when
a prompt is no longer worth extending from an estimate of the best score
its live beams can still reach. False estimates it from the current
length. True does the same and also stops recording completed beams once
every slot of the completed set is full. “never” estimates it from the
whole budget when the penalty rewards length.
The prompt is prefilled once and its cache row is copied into width
rows. Every step gathers each beam’s parent row through
DecodeOps.reindex, so a branched beam decodes exactly like a
separately selected prefix. Parameters are never mapped.
n is how many completed beams to return, which is separate from the
search width, and an n larger than width raises ValueError. A
selected path is a search result and not a draw, so its behaviour log
probability is zero. The raw log probabilities are still the model’s own
for the tokens on the path.
keep: int-
The number of continuations a step keeps, as
beams_to_keepupstream.
Generation
Section titled “Generation”class Generation( rows: int | None = struct.field(pytree_node=False, default=None), decoder: Callable[[ArrayLike, ArrayLike, int], tuple[str, ...]] | None = struct.field(pytree_node=False, default=None),)The prompt and padded continuation of each row, with response-aligned log-probabilities.
lengths counts each row’s response actions, including EOS.
terminated is true where a stopping criterion (EOS by default) ended
the row and false where the length limit did. Both log-probability
arrays have shape [B, max_new_tokens], and only positions below
lengths are valid. behavior_log_probs describes the distribution
that actually drew each action, after the whole transform chain.
raw_log_probs describes the unmodified model policy.
A request for n continuations per prompt gives every array
[B * n, ...] rows: prompt zero’s n continuations, then prompt
one’s. Each row has its own length, termination and likelihoods.
The arrays keep the placement the task ran with. On a mesh they are
global arrays whose rows are split over the batch axes and padded to the
device count. host() reads this process’s rows real rows back as
host arrays, and text decodes them through the task’s processor.
text: tuple[str, ...]-
Each real row’s valid continuation, decoded on first access (
row_texts).
Generation.host
Section titled “Generation.host”def host() -> Generation[np.ndarray]Return this process’s real rows as host arrays, without the padding rows that fill the devices.
Grammar
Section titled “Grammar”class Grammar()A token automaton on the device that keeps sampled text inside a regex or JSON schema.
dew.sampling.guided.regex and dew.sampling.guided.json_schema
compile one against a tokenizer’s vocabulary, and Sample(grammar)
applies it. transitions[state, class] is the state a token of that
class leads to, or -1 where the token is not allowed, and
classes[token] is the token’s class. State 0 is the state before the
first draw, so a zeroed carry starts a row.
Grammar.start
Section titled “Grammar.start”def start(rows: int) -> jax.ArrayGrammar.masked
Section titled “Grammar.masked”def masked(state: jax.Array, logits: jax.Array) -> jax.ArrayReturn logits [rows, vocab] with the tokens each row’s state forbids set to -inf.
Grammar.guiding
Section titled “Grammar.guiding”def guiding( transform: Callable[[StepState, jax.Array], jax.Array], state: jax.Array,) -> Callable[[StepState, jax.Array], jax.Array]Return transform, applied after masking the tokens each row’s state
forbids. A LogitsChain stays one, the mask its first transform, so a
terminal Greedy still draws by argmax (strategies.draw).
Grammar.advanced
Section titled “Grammar.advanced”def advanced(state: jax.Array, token: jax.Array, drawn: jax.Array) -> jax.ArrayReturn each row’s state after token; a row that did not draw keeps its state.
A drawn token that the state forbids can only come from a transform
that forces a token after the mask, such as ForcedEOS. Such a
token fails the device check, so the text never leaves the language.
Sample
Section titled “Sample”class Sample(grammar: Grammar | None = None)Draws every row independently, one token per step.
This is the loop generate runs when a request names no strategy. A
prompt’s continuations share its prefill (continuations).
grammar restricts every draw to a regex or JSON schema
(dew.sampling.guided). Each row keeps its own automaton state, and the
tokens the automaton forbids get the score -inf before the transform
chain runs. So the chain filters and samples within the language, and
the raw likelihood is still the model’s own.
Sampling
Section titled “Sampling”class Sampling( temperature: float = 1.0, top_k: int | None = None, eos_token_ids: int | tuple[int, ...] | None = None, pad_token_id: int | None = None, top_p: float = 1.0, min_p: float = 0.0, repetition_penalty: float = 1.0, presence_penalty: float = 0.0, frequency_penalty: float = 0.0, no_repeat_ngram_size: int = 0, min_new_tokens: int = 0, typical_p: float = 1.0, stop: tuple[str, ...] = (),)Common generation controls that decide how tokens are drawn and when a row stops.
Zero temperature is deterministic argmax, and top_k=None keeps the
whole vocabulary. EOS counts as a sampled action; the output slots after
it contain pad_token_id and have no likelihood.
The repetition penalty, no_repeat_ngram_size, min_new_tokens and
typical_p are Transformers’ controls of the same names, and
presence_penalty and frequency_penalty are vLLM’s. Each one’s
default changes nothing. stop ends a row whose text ends with one of
the strings; a task compiles them against its processor.
eos_token_ids and pad_token_id, Transformers’ names, belong to the
model and its tokenizer, so where a policy leaves them None, a task fills
in its own, as Transformers does. Plain generate has no task, so it
stops on EOS only when the policy names one, and pads with 0 unless the
policy names a pad_token_id. The block and masked decoders take the
same two names.
A Sampling value compiles to its transforms in Transformers’ order
when a request has no explicit logits chain. An explicit chain replaces
those transforms. The EOS and stop-string criteria still join the
request’s stopping criteria.
pad: int-
The id in the output slots after EOS, which is
pad_token_id, or 0 when it is None. stops: tuple[int, ...]-
The EOS ids that end a draw, or an empty tuple when the policy names no EOS.
Sampling.active
Section titled “Sampling.active”def active(names: Sequence[str]) -> list[str]Return the controls in names that this policy sets away from their defaults.
A client for a remote engine that cannot apply some controls the way Dew does passes their names here, and refuses a policy that sets any of them so the engine never draws without them.
Sampling.transforms
Section titled “Sampling.transforms”def transforms() -> tuple[LogitsTransform, ...]Return the full default chain that a request without explicit transforms runs.
At zero temperature the chain ends in the argmax and runs none of the
sampling-only filters, as Transformers does with do_sample=False.
To add a transform of your own, pass
logits=policy.transforms() + (mine,).
Sampling.criteria
Section titled “Sampling.criteria”def criteria() -> tuple[Stopping, ...]Return the EOS criterion this policy adds after a caller’s criteria, empty when it names no EOS.
Speculative
Section titled “Speculative”class Speculative( block: int = struct.field(pytree_node=False, default=4), confidence: float = struct.field(pytree_node=False, default=0.0),)Drafts with the model’s prediction depths or block drafter, and verifies with the model itself.
The acceptance rule is algorithm 1 of arXiv 2211.17192, as
_speculative_sampling in Transformers 5.16.1 applies it. The first
candidate of a block is an ordinary target draw, so it is always
accepted. The model’s prediction depths chain the remaining candidates
from the target’s last hidden state and each candidate’s embedding, as
vLLM’s Qwen3_5MultiTokenPredictor does. A block drafter (DeepSeek-V4.1’s
DSpark) instead drafts all the candidates after the first in one pass,
drawing each from its position’s logits as the pass reaches it.
A proposed token x is accepted with probability min(1, p(x) / q(x)),
where p is the target’s post-transform distribution and q is the
draft’s actual one, compared as a log ratio. At the first rejection, the
replacement is drawn from the normalized positive part of p - q, and a
block with nothing rejected draws a bonus token from p. So the emitted
tokens have exactly the distribution Sample would give them, token for
token, though not the same draws at one seed.
Every emitted token, including a replacement or a bonus, records the
target’s post-transform log probability as its behaviour log probability
and the model’s own log probability as its raw value. The draft’s q,
the acceptance probability and the residual are never recorded, because
none of them is the distribution the emitted token came from.
Each iteration proposes block candidates (at least 2), so every
collective keeps the same size. The target cache is saved before the
block and the accepted prefix is replayed into it, because a recurrent
mixer’s state is a running summary that no cursor can rewind. The
prediction cache is rebuilt the same way. A block that continues emits
two or more tokens unless the budget ends first, or one where a row walks
across a cache rebuild (block_step), in at most block blocks, so
ceil(budget / 2) iterations bound the loop, ceil((budget + block) / 2)
where the model’s caches rebuild.
confidence stops the draft after the first candidate to which the
draft itself gives a probability below confidence, as the reference’s
ConfidenceCriteria does. The later candidates are still computed, at
the same shapes, but cannot be accepted. Calling the strategy on a model
that has neither prediction depths nor a block drafter raises
ValueError.
Strategy
Section titled “Strategy”class Strategy(Protocol)The device loop that runs one generation request.
A strategy is called with the prefilled DecoderState, the initial
StepState, the DecodeOps that advance the model and gather its cache
rows, the composed transform chain, the stopping criterion, the token
budget and the number of continuations. It returns one Draws record
per output row. Sample, Beam and Speculative use nothing else, so
your own strategy is a callable with the same signature. Only
Speculative drafts, so only it is given the model’s prediction depths
and block drafter (DecodeOps.propose).
generate
Section titled “generate”def generate( model: nn.Module, params: Variables, inputs: ModelInputs | ArrayLike | Sequence[Sequence[int]], max_new_tokens: int, *, key: int | jax.Array, sampling: Sampling = _DEFAULT_SAMPLING, n: int = 1, logits: Transforms | None = None, stopping: Criteria | None = None, strategy: Strategy | None = None, trips: int | None = None, prefix: int | Literal['prompt'] | None = None, progress: Progress = END,) -> GenerationGenerate continuations from model inputs, or from token ids for a text-only prompt.
params is the full variables dict, {'params': ...}.
ModelInputs.token_fields["attention_mask"] marks the real tokens;
without a mask, every token is real. Every row must contain at least one
real token. Prefill evaluates the conditioning once, and decoding reuses
the cache and logical-position state the model keeps. Each cache
compacts the real input tokens and leaves paused rows unchanged.
Parameters keep their placement, and decode at the model’s compute
dtype: a call casts a parameter stored wider to it once, before its
loop (dew.interop.weights.decoding_storage), and holds that copy
beside the caller’s weights while it runs (fp32 weights under bf16
compute: half their bytes again). On a mesh, the rows are split over
its batch axes and the result keeps that sharding; Generation.host()
reads a process’s own rows back. All cooperating processes must use the
same input shapes, effective decoding components, padding id and
continuation count. The decode loops have fixed bounds, and the
processes agree on any block they skip. Keys fold in the global row
index and response position, so a pool draws what one process draws for
the same rows.
The n continuations of each prompt share its prefill and come back
as n consecutive rows of every array, in prompt order. Continuation
zero of a prompt draws with that prompt’s own key, so n=1 and
continuation zero of any larger request are the same draw.
logits is the whole transform chain, in the order it runs. None
means the chain sampling compiles to, and () runs no transform.
stopping adds criteria beside the policy’s EOS criterion and never
replaces it. strategy replaces the per-row draw loop; None uses
Sample. A policy with stop strings raises ValueError,
because generate has no tokenizer to compile them against.
trips, at least max_new_tokens, scans that many draws, so calls
of several budgets share one compiled loop (TextGeneration buckets
them); the budget still bounds what the request draws as far as a
transform or a search reads it (ForcedEOS, Beam), and the
columns past it come back for the caller to cut.
progress is the run’s progress a model that reads one draws at
(dew.nn.progress): its prefill’s and every step’s forward read it, an
input of the compiled loop like the weights, so a run’s collection draws
from its policy at the updates it committed; END, a finished run’s.
prefix serves a prefix run (LMObjective’s): each prompt that
carries no blocks of its own reads its first prefix real tokens, or
with "prompt" all of them, as one block (served_blocks). They are
made of each process’s own rows inside the request’s agreed check, and a
prompt with fewer real tokens than an int prefix is refused there, by
every process of a pool together.
sample
Section titled “sample”def sample( denoise: Denoiser | DiscreteDenoiser, x_T: jax.Array, steps: int | None = None, *, solver: Solver[StateT], guidance: Guidance | None = None, key: int | jax.Array | RowKeys, times: ArrayLike | Sequence[float] | None = None, initial: StateT | None = None, final_denoise: bool = True, record: Callable[[jax.Array, StateT, jax.Array], RecordT] | None = None,) -> jax.Array | tuple[jax.Array, RecordT]Run the reverse process from x_T with solver and return the final sample.
The time grid runs from T to 0. The solver takes one step across each
interval, and the result is the model’s clean prediction at the last
point. Pass exactly one of steps and times. With steps, the
process supplies its grid (times(steps)): that many points for a
Gaussian process, and MDLM’s grid of that many reveal steps for a masked
one (DiscreteProcess.times). An explicit times grid is
used as the trajectory, descending and concrete, for a source whose
solver pairs its own sigma and model-time tables. Its length sets the
number of steps, so a grid of steps + 1 points ending on the terminal
is allowed, and a single point takes no solver step.
final_denoise=False returns the last point’s state without the closing
clean prediction, which is how those solvers end.
A caller that compiles one walk for many grids of a length passes times
as data, traced, with initial, the solver’s state before the first step
that it built on the host from the concrete grid (solver.init), whose
checks and tables need the grid’s values.
record reads each step’s new state, the solver’s state after it and
the clean prediction the step took, and the sample comes back with those
readings stacked over the steps, (sample, records); Flow-GRPO records
each transition and its density this way (FlowSDE.trajectory), and a
figure of the walk each step’s clean prediction. A record needs at least
one step.
denoise is process.denoiser(...), which holds the process the solver
reads, and guidance wraps it. Every step’s noise comes from key
folded with the step index, so a trajectory is reproducible from one
key.
key may be RowKeys, one key a row, for a solver that draws row by row
(RowSolver, Unmask): each row’s draws are its own key’s, folded with the
step index, while the model still runs once over the batch. Another
solver refuses them by name.
The trajectory is one lax.scan, traced at each call. Under a caller’s
jax.jit it compiles once, which is how the pipelines and objectives
call it. Called eagerly, it traces and compiles the scan again every
time.
with_ids_of
Section titled “with_ids_of”def with_ids_of(policy: Sampling, defaults: Sampling) -> SamplingReturn policy with the EOS and pad ids it leaves None taken from defaults.
A task’s policy holds its model’s and tokenizer’s ids, so a call that replaces the policy keeps stopping and padding where the task does.