dew.nn.backbones.joint
The blocks shared by the published MM-DiT family: SD3, Flux, FLUX.2, Z-Image and Qwen-Image.
Each of these models is a stack of modulated residual blocks whose attention joins an image stream and a text stream, as Diffusers 0.34.0 to 0.40.0 run them. This module holds the shared parts: the adaLN modulation, the layer norm without affine parameters, one joint attention, the double-stream block that SD3, Flux and FLUX.2 run, the feed-forwards and the embedders. Each family’s own module keeps what only that family does.
| Name | Summary |
|---|---|
DoubleStreamBlock | Runs one double-stream block of SD3, Flux, FLUX.2 or Dew’s own MM-DiT. |
JointAttention | Runs the source’s Attention over an image stream and an optional context. |
embedding | Apply TimestepEmbedding or PixArtAlphaTextProjection to x. |
guided_time | Return the time’s embed, plus the guidance’s when the checkpoint embeds its distilled guidance. |
layer_norm | Return the source’s LayerNorm(elementwise_affine=False). |
modulate | Return the source’s norm(x) * (1 + scale) + shift for a normed x, broadcast over tokens. |
DoubleStreamBlock
Section titled “DoubleStreamBlock”class DoubleStreamBlock( context_first: bool = False, qk_norm: bool = True, bias: bool = True, epsilon: float = 1e-06, qk_epsilon: float | None = None, mlp_hidden: int | None = None, swiglu: bool = False, shared_modulation: bool = False, zero_modulation: bool = False, context_pre_only: bool = False, dual_attention: bool = False, rotary_pairs: Literal['half', 'adjacent'] = 'adjacent', dropout_rate: float = 0.0, dtype: Dtype | None = None, precision: PrecisionLike = None, attention_impl: AttentionImpl = 'auto', force_fp32_for_softmax: bool = True,)Runs one double-stream block of SD3, Flux, FLUX.2 or Dew’s own MM-DiT.
It matches SD3’s JointTransformerBlock, Flux’s FluxTransformerBlock
and FLUX.2’s Flux2TransformerBlock. The image and the context each
modulate and norm their own residual stream. They meet in one
JointAttention, and each adds its gated attention output back to its
stream. Then each runs its own modulated feed-forward and adds the gated
output. A block’s modulation is its own projection of the conditioning
vector (norm1, norm1_context): six pieces per stream, a shift, scale
and gate before the attention and again before the feed-forward.
dropout_rate drops each projected attention and feed-forward output
while train, before its gate.
context_first: bool-
Whether the context comes first in the joined attention, as in Flux and FLUX.2.
bias: bool-
Whether the projections have biases. FLUX.2’s have none.
qk_epsilon: float | None-
The query and key norms’ epsilon, None for
epsilon. Dew’s MM-DiT norms its streams at its stack’s epsilon and its queries and keys at 1e-6. mlp_hidden: int | None-
The feed-forward’s hidden width, four times the width for None.
swiglu: bool-
Whether the feed-forward is FLUX.2’s SwiGLU rather than the tanh GELU.
shared_modulation: bool-
Whether the conditioning is the model’s shared modulation, as in FLUX.2: an (image, context) pair of six pieces each, which every block reads.
zero_modulation: bool-
Whether each modulation starts at zero (
Modulation.zero_init). context_pre_only: bool-
Whether this is SD3’s last block, which reads the context and returns it unchanged. Its context norm then takes only a scale and a shift, and it has no context output projection or feed-forward.
dual_attention: bool-
Whether the block runs SD3.5’s second self-attention over the same normalized input, modulated by three more image pieces.
rotary_pairs: Literal['half', 'adjacent']-
The channels each angle turns (
JointAttention.rotary_pairs).
DoubleStreamBlock.model_flops
Section titled “DoubleStreamBlock.model_flops”def model_flops(lengths) -> floatDoubleStreamBlock.multisequence_flops
Section titled “DoubleStreamBlock.multisequence_flops”def multisequence_flops(lengths, context_lengths) -> floatThe declared attention and MLPs, plus each stream’s per-row modulation.
JointAttention
Section titled “JointAttention”class JointAttention( bias: bool = True, qk_norm: bool = True, epsilon: float = 1e-06, context_first: bool = False, project: bool = True, context_out: bool = True, rotary_pairs: Literal['half', 'adjacent'] = 'adjacent', dtype: Dtype | None = None, precision: PrecisionLike = None, attention_impl: AttentionImpl = 'auto', force_fp32_for_softmax: bool = True,)Runs the source’s Attention over an image stream and an optional context.
Each stream projects its own queries, keys and values; the context uses
the add_* projections. Per-head RMS norms normalize the queries and
keys, the rotation is applied to them, and one softmax attends over the
joined sequence. The result is split back into the two streams, and each
goes through its own output projection. SD3’s JointAttnProcessor2_0
puts the image first in the joined sequence, and Flux’s processor puts
the context first. Without a context, it is self-attention over the
image. The rotation is the cosines and sines of every joined token, one
angle per channel pair, [B or 1, S, D // 2], or of the image’s tokens
alone, which turn before the join and leave the context as it is (Dew’s
MM-DiT); lengths is the number of real keys in each row.
bias: bool-
Whether every projection has a bias. FLUX.2’s, Z-Image’s and Qwen-Image’s have none.
qk_norm: bool-
Whether the queries and keys are RMS-normed. SD3 without a
qk_normconfig leaves them as they are. context_first: bool-
Whether the context’s tokens come first in the joined sequence, as in Flux.
project: bool-
Whether the attention applies its own output projection. Flux’s single-stream block projects the attention output itself, together with its feed-forward, so it takes the output unprojected.
context_out: bool-
Whether the context’s output is projected and returned. SD3’s last block reads no context back, so it has no
to_add_out. rotary_pairs: Literal['half', 'adjacent']-
The channels each angle turns (
dew.nn.rope.rotate): the published families’ adjacent pairs, or the rotate-half halves of Dew’s own MM-DiT.
JointAttention.model_flops
Section titled “JointAttention.model_flops”def model_flops(lengths) -> floatJointAttention.stream_flops
Section titled “JointAttention.stream_flops”def stream_flops( lengths, context_lengths, image_features: int, context_features: int | None = None,) -> floatBoth streams’ Q/K/V maps and joined QK/AV, with optional output maps.
JointAttention.attend
Section titled “JointAttention.attend”def attend(query, key, value, lengths)Return one softmax attention over the joined sequence, masking each row’s keys past lengths.
embedding
Section titled “embedding”def embedding( owner: nn.Module, x, features: int, name: str, *, bias: bool = True, dtype: Dtype | None = None, precision: PrecisionLike = None, activation: Callable[[jax.Array], jax.Array] = nn.silu, sharded: bool = True,)Apply TimestepEmbedding or PixArtAlphaTextProjection to x.
Both are a linear layer, an activation and a second linear layer. The
activation is SiLU, or Wan’s tanh GELU. The two layers are stored flat
as {name}_linear_1 and {name}_linear_2 in the calling module, each
sharding its output, the model’s width, where sharded; a model that
names no axes of them (the UNet’s) leaves them to the shape heuristic.
guided_time
Section titled “guided_time”def guided_time( embed: Callable[[jax.Array, str], jax.Array], time, guidance, *, guidance_embeds: bool,)Return the time’s embed, plus the guidance’s when the checkpoint embeds its distilled guidance.
The guidance is multiplied by a thousand before embedding, as the source
does. A guidance-embedded checkpoint requires guidance, and any other
checkpoint raises ValueError when guidance is given.
layer_norm
Section titled “layer_norm”def layer_norm(dtype, epsilon: float = 1e-06)Return the source’s LayerNorm(elementwise_affine=False).
Its eps is 1e-6, the value SD3 and Flux fix, unless a config gives another.
modulate
Section titled “modulate”def modulate(x, shift, scale)Return the source’s norm(x) * (1 + scale) + shift for a normed x, broadcast over tokens.