Skip to content

dew.diffusion.discrete

Masked (absorbing-state) discrete diffusion, with the interface of the Gaussian one.

The forward process replaces each token with a mask id independently, with a probability that grows with t in [0, 1]. MaskingSchedule.alpha(t) is the fraction of tokens still visible at t. Training minimizes the continuous-time negative ELBO of MDLM (Sahoo et al. 2024, “Simple and Effective Masked Diffusion Language Models”), which is the cross entropy of the model’s prediction at the masked positions, weighted by -alpha’(t) / (1 - alpha(t)).

Sampling reverses the process one interval at a time. A masked token is revealed with probability (alpha(s) - alpha(t)) / (1 - alpha(t)), and a revealed token is drawn from the model’s categorical distribution. This is MDLM’s _ddpm_update.

DiscreteProcess provides what dew.sampling.sample needs from a process: a time grid, an initial state and a denoiser. The denoiser’s two outputs sit where a Gaussian denoiser returns x_0 and epsilon. They are the model’s argmax fill of the masked positions and the log-probabilities the solver draws from.

NameSummary
MDLMBuilds MDLM’s masked diffusion process on the log-linear schedule.
ConfidentLLaDA2’s reveal, as its release’s generate takes it (inclusionAI/LLaDA2.0-mini@dad945c, modeling_llada2_moe.py): each step draws every masked position’s token from the model’s distribution at temperature, its top_k best kept, and reveals the positions whose drawn token holds more than threshold of that distribution where they number at least the step’s quota, else the quota’s most probable, ties taken in slot order.
DiscreteDenoiserMaps (x_t, t) to (argmax fill, log-probabilities) for model under params.
DiscreteProcessMasks tokens of a vocabulary whose mask token is mask_id.
LogLinearMDLM’s log-linear schedule, alpha(t) = 1 - (1 - eps) t.
MaskingScheduleSets how fast tokens are masked as t grows.
UnmaskIntegrates a DiscreteProcess with MDLM’s reverse step from t to s < t.
alignedscores [B, S, ...] per position as the model’s predictions of each position’s token: its own for a model whose logits line up with its inputs (LLaDA’s, MDLM’s), and for one that declares shifted_logits (Dream-v0’s) the position before’s, as its reference shifts them.

dataclass source

class MDLM(mask_id: int, eps: float = 0.001, endpoint: float | None = None)

Builds MDLM’s masked diffusion process on the log-linear schedule.

MDLM is Sahoo et al. 2024.

endpoint: float | None

The time the sampling grid runs to (DiscreteProcess.endpoint), None for MDLM’s.

dataclass source

class Confident(
threshold: float = 0.95,
temperature: float = 1.0,
top_k: int | None = None,
)

LLaDA2’s reveal, as its release’s generate takes it (inclusionAI/LLaDA2.0-mini@dad945c, modeling_llada2_moe.py): each step draws every masked position’s token from the model’s distribution at temperature, its top_k best kept, and reveals the positions whose drawn token holds more than threshold of that distribution where they number at least the step’s quota, else the quota’s most probable, ties taken in slot order. The quota spreads the walked block’s slots, its prompt’s among them (DiscreteDenoiser.block; the slots masked at the walk’s first step where no block is walked), over the walk’s steps as evenly as they divide, the first steps one more, so a block is whole by its last step. The mask token is no draw (DiscreteDenoiser).

Two choices are Dew’s where the release’s code does otherwise: temperature 0 takes each position’s best token, as Dew’s samplers do, where the release’s generate documents greedy decoding but still draws, unscaled; and a tie at the quota’s edge goes to the earlier slot, which torch.topk leaves unspecified. At top_k 1, where each position’s best token is unique and the threshold leaves no tie at the quota’s edge, the two agree, which tests/test_blockwise_walk.py holds over its recording. A step with nothing masked reveals nothing and draws nothing.

def init(x, times, process, *, key) -> tuple[jax.Array, jax.Array, jax.Array]

The step reached, the walk’s step count, and each row’s slots the quota spreads (set at the first step, -1 until then). A process that masks no tokens is refused, as Unmask refuses it.

def step(x, t, t_next, denoised, log_probs, state, key, process, denoise)
def step_rows(x, t, t_next, denoised, log_probs, state, keys, process, denoise)

step, row i drawing its tokens from keys[i] alone (RowSolver).

dataclass source

class DiscreteDenoiser(
process: DiscreteProcess,
model: nn.Module,
params: Variables,
inputs: ModelInputs | None = None,
mutable_mask: jax.Array | None = None,
block: jax.Array | None = None,
visible: jax.Array | None = None,
)

Maps (x_t, t) to (argmax fill, log-probabilities) for model under params.

The denoiser ignores t, because the masked model is conditioned on the corruption it sees and not on the time. Unmask.step reads the time from the process. t stays in the signature because sample calls every denoiser as (x_t, t). The model’s full-sequence logits are read through its logits, or a native module’s own forward (forward_method).

The model’s own logits at an unmasked position do not matter, since the position keeps its token (MDLM’s carry-over parameterization). The mask token gets zero probability, whatever score the model gives it, because it marks corruption and a revealed position must never draw it. When mutable_mask is given, only masked positions inside it can change.

A blockwise walk (DiscreteProcess.generate’s block) names the block it walks, block [B, S], its prompt slots among them, which a quota of reveals spreads over (Confident), and the keys each position reads, visible [B, S, S]: its document’s real slots through its own block, which the model takes as the call’s attention_pairwise_mask. Once the walked block is revealed, a step runs no model.

def masked(tokens)

dataclass source

class DiscreteProcess(
schedule: MaskingSchedule,
mask_id: int,
endpoint: float | None = None,
)

Masks tokens of a vocabulary whose mask token is mask_id.

endpoint: float | None

The time a release’s sampling grid runs to (Dream’s releases’ eps): times walks linspace(T, endpoint, steps + 1) with its last point taken to 0, where an Unmask step reveals every position still masked, as their last step does (p_transfer = 1); None is MDLM’s grid to NOISE_REMOVAL_TIME and its closing fill.

T

The fully masked end of the time domain, named T as in a Gaussian Process.

def sample_t(key, n: int) -> jax.Array

Return n training times stratified over [SAMPLING_EPS, 1), MDLM’s antithetic draw.

Row i draws its own point in the i-th of n strata, so the weights 1 / t of one batch cover the whole trajectory. The floor at SAMPLING_EPS keeps every weight at most 1 / SAMPLING_EPS.

def stratified(uniforms, strata, n: int) -> jax.Array

sample_t from a uniform for each row and its stratum of n.

def corrupt(key, tokens, t) -> tuple[jax.Array, jax.Array]

Return (masked tokens, is_masked) at t, with one t per row.

def masked(draws, tokens, t) -> tuple[jax.Array, jax.Array]

corrupt from a uniform draw for each of the tokens: those under 1 - alpha(t) masked.

def weight(t) -> jax.Array

Return the NELBO weight -alpha’(t) / (1 - alpha(t)) on the masked cross entropy.

The weight is exactly zero at t = 0. Nothing is masked there, so no token contributes, and the quotient itself is undefined.

def times(steps: int) -> jax.Array

MDLM’s sampling grid for steps reveal steps: steps + 1 times from T to NOISE_REMOVAL_TIME, at which sample’s closing argmax fills what is still masked, as MDLM’s noise removal does; with an endpoint, the release’s grid to it, its last point 0.

def noise(key, shape) -> jax.Array

Return x_T, with every position masked.

key is not used, because the fully masked state is a single point and needs no random draw.

def denoiser(
model: nn.Module,
params: Variables,
conditions: Mapping[str, Conditioning] | None = None,
unconditional: Mapping[str, Conditioning] | None = None,
*,
inputs: ModelInputs | None = None,
mutable_mask: jax.Array | None = None,
) -> DiscreteDenoiser
def generate(
model: nn.Module,
variables: Variables,
inputs: ModelInputs | jax.typing.ArrayLike,
max_new_tokens: int,
*,
key: int | jax.Array,
n: int = 1,
steps: int = MDLM_STEPS,
solver: Unmask | Confident | None = None,
eos_token_ids: tuple[int, ...] = (),
pad_token_id: int = 0,
block: int | None = None,
) -> CanvasGeneration

Generate max_new_tokens tokens after each prompt with native MDLM.

The whole response span starts masked, and solver (Unmask by default) unmasks it over a grid of steps times. Each prompt row gets n continuations in the returned CanvasGeneration. Prompt tokens never change, including literal mask ids. An EOS token trims the finished response, and the tokens after it become pad_token_id, but it does not end the bidirectional refinement early.

The response continues the prompt’s token fields (ModelInputs.extended): its slots are text, at the positions after the prompt’s, and the conditioning is the prompt’s own.

With block, the response is walked a block at a time, as LLaDA2’s releases generate (their generate(block_length=)): the row’s tokens cut into blocks of block places, counted over its document’s real tokens from its first (dew.nn.inputs.numbered_blocks), and each block holding response slots unmasked over its own steps with every other slot held, in order, each position reading its own block’s keys and the earlier blocks’ alone (the call’s attention_pairwise_mask), so a block sees the clean blocks before it and none after. The last block is walked whole, past max_new_tokens where it ends inside it, and the response kept to max_new_tokens; the canvas holds that block’s block - 1 slots past the response at most, so the model’s length must too. A row stops once its response holds an end token: a block after the one that revealed it is not walked. A block walks at most max_new_tokens steps, as the release caps its steps.

Raises ValueError when the model does not read the whole row (refuse_causal), when the prompt plus max_new_tokens exceeds the max_seq_len it declares, or when the inputs include a token field a response has no rule to continue.

dataclass source

class LogLinear(eps: float = 0.001)

MDLM’s log-linear schedule, alpha(t) = 1 - (1 - eps) t.

The masking rate -log alpha is then linear in log space and the NELBO weight is 1 / t.

def masked_fraction(t)
def alpha_prime(t)

class source

class MaskingSchedule(ABC)

Sets how fast tokens are masked as t grows.

masked_fraction(t) in [0, 1) is the fraction of tokens masked at t, zero at t = 0, which each schedule states in a form exact near zero: the NELBO weight divides by it, and 1 - alpha(t) there keeps only the last bits of alpha’s rounding. alpha(t), the fraction left unmasked, is its complement.

def masked_fraction(t) -> jax.Array
def alpha_prime(t) -> jax.Array

Return d alpha / dt, which is negative.

def alpha(t) -> jax.Array

dataclass source

class Unmask(temperature: float = 1.0)

Integrates a DiscreteProcess with MDLM’s reverse step from t to s < t.

Each masked position is revealed with probability (alpha(s) - alpha(t)) / (1 - alpha(t)), and a revealed position takes a token drawn from the model’s categorical distribution at temperature, its best token at 0 (Dream’s releases’ alg='origin' at their default temperature 0, sample_tokens). The rest stay masked, as does a position whose scores hold no distribution (a NaN), so a walk never commits an arbitrary token: generation refuses a response left masked. Stepping any other kind of process raises ValueError.

def init(x, times, process, *, key) -> tuple
def step(x, t, t_next, denoised, log_probs, state, key, process, denoise)
def step_rows(x, t, t_next, denoised, log_probs, state, keys, process, denoise)

step, row i drawing its reveals and tokens from keys[i] alone (RowSolver), as the row sampled by itself draws them.

function source

def aligned(
model: nn.Module,
scores: jax.Array,
fields: Mapping[str, jax.Array | np.ndarray | Coordinates] | None = None,
) -> jax.Array

scores [B, S, ...] per position as the model’s predictions of each position’s token: its own for a model whose logits line up with its inputs (LLaDA’s, MDLM’s), and for one that declares shifted_logits (Dream-v0’s) the position before’s, as its reference shifts them. A position that opens its text, the row’s first, the first real one after padding and each packed document’s first in the token fields’ layout (token_layout), reads its own, as the reference repeats a row’s first: no document reads another’s.