dew.objectives.rl.nft
DiffusionNFT: online diffusion reinforcement on the forward process (arXiv 2509.16117).
The policy is trained by flow matching on clean samples its old self drew, each sample’s target split between an implicit positive and negative policy by an optimality probability its reward sets (the paper’s Eq. 5): with v+ = (1 - beta) v_old + beta v_theta and v- = (1 + beta) v_old - beta v_theta, a sample of optimality r trains r on v+ and 1 - r on v-. The sampler is any solver over the old policy, no likelihood is computed and no classifier-free guidance is used.
The loss is the released trainer’s form (NVlabs/DiffusionNFT at cbb14f84, scripts/train_nft_sd3.py
:862-970), which the paper’s Eq. 5 is up to a weighting: each implicit policy’s clean-sample estimate,
x0 = x_t - t v, against the sample, its squared error over the sample’s coordinates divided by the
detached mean absolute error (floored at 1e-5), the two mixed by r, divided by beta and scaled by the
advantage clip; plus kl (the release’s train.beta, 1e-4) times the mean squared difference of the
live velocity from the base model’s (:962-967). An advantage is the reward less its prompt group’s
mean over the global standard deviation of the batch’s rewards plus 1e-4 (flow_grpo/stat_tracking.py
:14-30, global_std); r is
that advantage clipped to the clip and mapped to [0, 1] (train_nft_sd3.py:910-929). The old policy
is the EMA slot, which follows the trained weights at old_decay once an update
(train_nft_sd3.py:1033-1038, the release’s decay_type 1), and samples every rollout (:640-660); the
base model is the starting tree’s weights, frozen under TEACHER. The release also keeps an EMA of
the trained weights at 0.9 for its evaluations and saved checkpoints (:565-567, :220, :342); Dew’s one
EMA slot holds the old policy, so evaluation, previews and the published task use the live weights.
| Name | Summary |
|---|---|
DiffusionNFTObjective | Trains a rectified-flow velocity model by DiffusionNFT (the module docstring). |
NFTRollout | Draws groups clean samples of each source row with the old policy, the objective’s solver over steps time points and no guidance (the release’s deterministic second-order DPM-Solver at guidance 1), and trains each at the first train_steps times of that grid (the release’s int(0.99 num_steps), 9 of its 10 steps’ times, at 11 time points). |
released_old_decay | The released trainer’s old-policy decay at the update after updates completed ones, min(0.001 s, 0.5) with s the gradient steps taken once it is made (train_nft_sd3.py:169-189 and 1033-1038, decay_type 1, one gradient step an epoch): 0.001 at the first update, a half from the 500th. |
released_policy_loss | Each sample’s policy loss [N] at one time, from its clean value samples, the live and old policies’ clean-sample estimates there (live, old, x_t - t v) and its optimality r: the implicit positive and negative policies’ estimates, beta live + (1 - beta) old and (1 + beta) old - beta live, each against the sample, its mean squared error over the sample’s coordinates divided by its detached mean absolute error floored at 1e-5, mixed r to 1 - r, divided by beta and times scale, the advantage clip (train_nft_sd3.py:934-960). |
DiffusionNFTObjective
Section titled “DiffusionNFTObjective”class DiffusionNFTObjective( model: nn.Module, process: Process | Preset, inputs: InputSpec, *, beta: float = 1.0, adv_clip_max: float = 5.0, kl: float = 0.0001, old_decay: optax.Schedule = released_old_decay, autoencoder: AutoEncoder | None = None, solver: Solver | None = None, steps: int = 41, variables: Variables | None = None, grid: SourceGrid | None | Omitted = OMITTED,)Trains a rectified-flow velocity model by DiffusionNFT (the module docstring).
A batch holds latents, the clean samples the old policy drew, [N, ...]; timesteps, the
times each trains at, [N, T] (the sampler’s grid); and advantages, [N]. beta is the
implicit policies’ strength, adv_clip_max the advantage clip and the loss’s scale, kl the
weight of the velocity’s squared difference from the base model’s, which a positive weight reads
from the starting tree (variables, a pretrained model’s), and old_decay the old policy’s decay
over updates. The release’s single-reward defaults: beta 1, a clip of 5, kl 1e-4. The
old policy is the EMA slot; evaluation, previews and the published task use the live weights.
solver and steps configure evaluation; a rollout samples over its own grid (NFTRollout).
DiffusionNFTObjective.held_variables
Section titled “DiffusionNFTObjective.held_variables”def held_variables(*, beside: Variables | None = None) -> VariablesDiffusionNFTObjective.optimality
Section titled “DiffusionNFTObjective.optimality”def optimality(advantages: jax.Array) -> jax.ArrayEach sample’s optimality probability r: its advantage clipped to adv_clip_max and mapped
linearly onto [0, 1], a zero advantage at one half (train_nft_sd3.py:910-929).
DiffusionNFTObjective.loss
Section titled “DiffusionNFTObjective.loss”def loss(variables: Variables, batch: Batch, step: Step) -> tuple[Ratio, Aux]NFTRollout
Section titled “NFTRollout”class NFTRollout( objective: DiffusionNFTObjective, reward: FlowReward, groups: int = 4, steps: int = 11, train_steps: int = 9,)Draws groups clean samples of each source row with the old policy, the objective’s solver
over steps time points and no guidance (the release’s deterministic second-order DPM-Solver at
guidance 1), and trains each at the first train_steps times of that grid (the release’s
int(0.99 num_steps), 9 of its 10 steps’ times, at 11 time points).
A sample’s advantage is its reward less its group’s mean, over the population standard deviation of every reward in the batch plus 1e-4 (the module docstring); the rewards stay float64 on the host and the advantages are float32.
released_old_decay
Section titled “released_old_decay”def released_old_decay(updates: jax.typing.ArrayLike) -> jax.ArrayThe released trainer’s old-policy decay at the update after updates completed ones,
min(0.001 s, 0.5) with s the gradient steps taken once it is made (train_nft_sd3.py:169-189 and
1033-1038, decay_type 1, one gradient step an epoch): 0.001 at the first update, a half from the
500th.
released_policy_loss
Section titled “released_policy_loss”def released_policy_loss( samples: jax.Array, live: jax.Array, old: jax.Array, optimality: jax.Array, *, beta: float, scale: float,) -> jax.ArrayEach sample’s policy loss [N] at one time, from its clean value samples, the live and old
policies’ clean-sample estimates there (live, old, x_t - t v) and its optimality r: the
implicit positive and negative policies’ estimates, beta live + (1 - beta) old and
(1 + beta) old - beta live, each against the sample, its mean squared error over the sample’s
coordinates divided by its detached mean absolute error floored at 1e-5, mixed r to 1 - r,
divided by beta and times scale, the advantage clip (train_nft_sd3.py:934-960).