dew.nn.diffusion_gemma
DiffusionGemma’s shared encoder and decoder, and its self-conditioning MLP.
The MLP follows the Transformers implementation: a scaled pre-norm, a gated feed-forward and a post-norm without scale. The previous step’s logits become soft embeddings through an fp32 softmax against the scaled embedding table. On the first inference step, an explicit self-conditioning mask zeros those embeddings. The official SFT objective supplies zero logits for its dropout branch instead, and zero logits give uniform soft embeddings, which are not a zero signal.
| Name | Summary |
|---|---|
DiffusionGemma | Reads one text parameter tree causally for the context and bidirectionally for canvases. |
SelfConditioning | Folds the previous step’s soft embeddings into the canvas embeddings: a GeGLU feed-forward (GatedMLP, its gate, up and down projections at this module’s own scope) of the normed signal, added to the embeddings and normed again. |
DiffusionGemma
Section titled “DiffusionGemma”class DiffusionGemma(DecoderWrapper)Reads one text parameter tree causally for the context and bidirectionally for canvases.
encode appends clean tokens to the cache. __call__ refines a canvas
against that frozen cache and feeds the previous logits through
self-conditioning. hidden_states and logits read clean tokens
through the same causal encoder over the whole sequence, with no cache.
Each method is a separate apply. The encoder and the decoder share one
scope, so their parameters are identical and no second tree is stored.
The training and cache hooks are its text model’s (DecoderWrapper), which the decoder
clones: the decoder shares text’s scope, another reader, not another bank, and causal is
the encoder’s, which setup holds causal.
DiffusionGemma.setup
Section titled “DiffusionGemma.setup”def setup()DiffusionGemma.with_trainable_layer_scalars
Section titled “DiffusionGemma.with_trainable_layer_scalars”def with_trainable_layer_scalars() -> DiffusionGemmaReturn this model with its layer scalars as parameters, the model the published SFT trains.
A source whose scalars are constants (“frozen”) is read by it through
trainable_variables. A model without layer scalars is refused.
DiffusionGemma.trainable_variables
Section titled “DiffusionGemma.trainable_variables”def trainable_variables(variables: VariableDict) -> VariableDictReturn this model’s variables as with_trainable_layer_scalars() reads them.
Google’s release makes each layer’s scalar a parameter, and
Transformers declares the same tensor a buffer, which a “frozen”
model keeps under constants. Each moves to its layer under params,
constants it leaves empty are dropped, and no array is copied. A
“trainable” model’s tree is returned as it is. A frozen model’s tree
that already holds a trainable scalar is refused.
DiffusionGemma.init_cache
Section titled “DiffusionGemma.init_cache”def init_cache(batch_size: int)DiffusionGemma.encode
Section titled “DiffusionGemma.encode”def encode( tokens, *, coordinates: Coordinates | None = None, image_indices=None, image_groups=None, block_groups=None, rotary_positions=None, attention_pairwise_mask=None, attention_key_positions=None, conditioning: Mapping[str, jax.Array] | None = None, train: bool = False, states: bool = False, progress: Progress | None = None,)Append a clean prompt or committed canvas to the cache, evaluating media only when given.
It returns the logits, or with states the final normalized states
before the head. A loss that scores the vocabulary a tile at a time
reads the states, so the vocabulary-sized logits of a whole row never
exist at once. The tokens are laid out by their coordinates, at a
training step’s progress (dew.nn.progress; None at the run’s end).
DiffusionGemma.hidden_states
Section titled “DiffusionGemma.hidden_states”def hidden_states( tokens, *, train: bool = False, image_indices=None, conditioning: Mapping[str, jax.Array] | None = None, **fields={},)Return the causal encoder’s final normalized states over clean tokens, writing no cache.
The encoder reads the text tree and fuses media as encode does, but
over the whole sequence at once and without the cache encode fills
for a canvas to attend to: the read a loss or a probe takes of clean
text. fields go to CausalTransformer.hidden_states.
DiffusionGemma.logits
Section titled “DiffusionGemma.logits”def logits(tokens, *, train: bool = False, **fields={})Return the causal encoder’s fp32 logits over clean tokens, writing no cache (hidden_states).
DiffusionGemma.logits_from_hidden
Section titled “DiffusionGemma.logits_from_hidden”def logits_from_hidden(hidden)Return the logits of final states hidden, through the head the encoder and the decoder share.
DiffusionGemma.output_table
Section titled “DiffusionGemma.output_table”def output_table() -> OutputTable | NoneReturn the shared head as the matrix final states contract, or
None where none does (CausalTransformer.output_table).
DiffusionGemma.vocabulary_rows
Section titled “DiffusionGemma.vocabulary_rows”def vocabulary_rows() -> jax.ArrayThe shared head’s rows (CausalTransformer.vocabulary_rows).
SelfConditioning
Section titled “SelfConditioning”class SelfConditioning( norm_eps: float = 1e-06, dtype: Dtype | None = None, precision: PrecisionLike = None,)Folds the previous step’s soft embeddings into the canvas embeddings: a GeGLU feed-forward
(GatedMLP, its gate, up and down projections at this module’s own scope) of the normed signal,
added to the embeddings and normed again.
SelfConditioning.setup
Section titled “SelfConditioning.setup”def setup()