dew.nn.vision
Vision towers, projectors and reference checkpoint maps.
transformers 5 ships no Flax classes, so the towers are vendored the way
dew/nn/text_encoders.py vendors CLIP, in the reference layout, with each
weight read from the checkpoint’s safetensors under its reference tensor name.
The operation order follows transformers 5.16.1
models/siglip/modeling_siglip.py, models/llama4/modeling_llama4.py,
models/gemma4/modeling_gemma4.py, models/qwen3_5/modeling_qwen3_5.py and
models/muse_glimmer/modeling_muse_glimmer.py, MiMo-V2.6’s modeling_mimo_v2.py and LFM3-VL’s
modeling_lfm3_vl.py.
The SigLIP trunk is patch convolution with bias, learned position embeddings
with no class token, pre-norm encoder blocks and a post layer norm. Its block
shares both halves with CLIP’s: the attention and the feed-forward
(dew.nn.text_encoders), the MLP carrying the config’s activation. The Llama 4
trunk is MetaCLIP-style: an unfold patch embedding without bias, a class token
appended after the patches, learned positions, a pre norm, full-attention
blocks with a complex rotary over the patch grid, a post norm, the class
token dropped, and the pixel-shuffle MLP inside the tower where the reference
keeps it. The Gemma 4 trunk is patch pixels scaled to [-1, 1] through a
bias-free map, summed 2D position tables, RMS-normed blocks with a 2D rotary
and gated feed-forwards, and a position pooler with standardization. The
Qwen 3.5, MiMo-V2.6, Muse Glimmer and LFM3-VL trunks are configs of one windowed and
full vision transformer (vit): a NaViT-style patchify with the still frame
repeated along time, an optional resampled position table, decoder blocks
with a 2D rotary that attend over a frame, a band of keys or a window of
patches, or convolve the frame’s patch grid (LFM3-VL’s), each layer in its own token order, and
the merge-block projector. Each tower’s projector is a
value beside it: Gemma 3’s averages each patch block, norms and maps to the
decoder width, Llama 4’s maps the shuffled output to the decoder width,
Gemma 4’s norms without a scale and maps, and Qwen 3.5’s is the merger.
Gemma 3n uses the MobileNet-v5 encoder in dew.nn.mobilenet and its own hard/soft
vision embedder. DeepSeek-V4.1’s trunk (the release’s
inference/vision.py) is a biased patch map, RMS-normed blocks with the
same 2D rotary as Qwen 3.5’s and a SwiGLU feed-forward, and a final norm; its
projector is the aligner, which groups squares of the patch grid through an
exact-GELU MLP and lays the image out as its decoder reads it.
Each family lives in its own module of this package (siglip, llama4,
gemma4, qwen35, mimo, muse_glimmer, lfm3, deepseek_v41, gemma3n) over the
bases in common and the shared trunk in vit,
with its path map and its config and weight translators, and each family’s
classes say where their tensors sit (TowerLayout, ProjectorLayout), so the
kinds dew.registry names are the one list of them. The package exports the
classes a caller constructs or subclasses, and reads a kind’s tensors through
its class’s layout (tower_variables, projector_variables). Each name is imported from its own module
when first read, so a caller of one family’s tower reads no other family’s.
| Name | Summary |
|---|---|
DeepseekV41Projector | DeepSeek-V4.1’s aligner and image span (vision.py:106-119, model.py:1228-1239). |
DeepseekV41Vision | DeepSeek-V4.1’s ViT geometry, under its vision_config’s names (vision.py:87-103). |
Gemma3nProjector | Gemma3nMultimodalEmbedder’s vision hard and soft token paths. |
Gemma3nVision | Gemma 3n’s vision tower: MobileNet-v5’s encoder, by its construction fields (MobileNetV5). |
Gemma4Projector | Soft tokens to text width: a scale-free RMS norm, then the map. |
Gemma4Vision | A Gemma 4 trunk’s geometry, under the reference’s field names. |
Gemma4VisionTransformer | The Gemma 4 vision trunk, param layout of Gemma4VisionConfig. |
GemmaProjector | Patch features into soft tokens: block average, norm, map to text width. |
Lfm3VlProjector | LFM3-VL’s projector (Lfm3VlMultiModalProjector) is MergeProjector: a layer norm of each trunk token at projector_layer_norm_eps, its bias kept whatever projector_bias says, each merge block’s tokens joined unit by unit, and two maps, biased by projector_bias, around the projector_hidden_act activation. |
Lfm3VlVision | LFM3-VL’s trunk geometry under its vision_config’s names. |
Llama4Projector | Tower output to text width: one bias-free map. |
Llama4Vision | A Llama 4 trunk’s geometry, under the reference’s field names. |
Llama4VisionTransformer | The Llama 4 vision trunk, param layout of Llama4VisionConfig. |
MergeProjector | Each merge block’s features as one vector, mapped to the decoder width. |
MiMoProjector | MiMo’s merger (MiMoVisionPatchMerger): a layer norm, each merge block grouped, and two maps around an exact GELU to the decoder width. |
MiMoVision | MiMo ViT’s geometry under its vision_config’s names. |
MuseGlimmerProjector | The merged image tokens as the decoder reads them (MuseGlimmerModel.get_image_features): each merge block’s features channel by channel through the adapter’s two bias-free maps, each followed by the activation (MergeProjector), a bias-free map to the decoder width, and a scale-free RMS norm (perception_emb_norm). |
MuseGlimmerVision | Muse Glimmer’s trunk geometry under its vision_config’s names. |
ProjectorBase | A native projector module whose constructor is its record schema. |
ProjectorLayout | Where one projector kind’s tensors sit in a media checkpoint and the leaves they fill: prefix before every name (or several, whose tensors each keep the name past their own), tensors each name’s leaf path, strip a further prefix a name may carry (a merger’s), and read the projector’s own reader where it checks its tensors (placed otherwise). |
Qwen35Projector | Qwen 3.5’s projector fields: the trunk width, the merge size, the merged width the text embeddings read directly. |
Qwen35Vision | A Qwen 3.5 trunk’s geometry, under the reference’s field names. |
SiglipVision | A SigLIP trunk’s geometry, under the reference’s field names. |
TowerBase | One tower kind’s value: its fields, how it builds its module, and where its tensors sit in a media checkpoint (checkpoint, None for a tower read another way, as the audio towers are). |
TowerGeometry | The shapes a conditioner has to invent to create a tower’s media leaves. |
TowerLayout | Where one tower kind’s tensors sit in a media checkpoint and the leaves they fill: prefix before every name, and path from a name past it to its leaf, rooted at its collection (params, and Gemma 4’s constants beside it). |
ViTLayout | The geometry and wiring of one family’s tower. |
VisionTransformer | A family’s trunk (ViTLayout): patches mapped and placed, the blocks, and the trunk sequence in merge-block order. |
projector_layout | Where projector kind kind’s tensors sit: its class’s checkpoint (dew.registry.projectors). |
projector_variables | One projector kind’s parameters, in the requested storage (projector_layout), and the layouts that write its tensors back from them. |
tower_layout | Where tower kind kind’s tensors sit: its class’s checkpoint (dew.registry.towers). |
tower_variables | One vision tower’s variables, in the requested storage (tower_layout), and the layouts that write its tensors back from them. |
DeepseekV41Projector
Section titled “DeepseekV41Projector”class DeepseekV41Projector(ProjectorBase)DeepSeek-V4.1’s aligner and image span (vision.py:106-119, model.py:1228-1239).
The patch grid is zero-padded at its bottom and right to whole
downsample_ratio squares, each square’s features concatenate
channel-major (unfold’s order), and a two-layer exact-GELU map takes
them to the decoder width. The span the decoder reads is a learned start
vector, each row of aligned features followed by a learned newline
vector, and a learned end vector: rows * (columns + 1) + 2 positions.
DeepseekV41Vision
Section titled “DeepseekV41Vision”class DeepseekV41Vision( num_hidden_layers: int = 32, hidden_size: int = 1024, num_attention_heads: int = 16, intermediate_size: int = 2816, patch_size: int = 14, rope_theta: float = 10000.0,)DeepSeek-V4.1’s ViT geometry, under its vision_config’s names (vision.py:87-103).
Its layout (ViTLayout): each patch_size square’s pixels, channel-major, through one biased
map; blocks of full attention whose queries, keys and values come from one biased map, rotated by
the grid’s rows then columns in fp32 as Qwen 3.5’s vision rope turns them (vision.py:8-21), then
a SwiGLU whose gate and up maps are one, each after an RMS norm (vision.py:46-84); a final RMS
norm, and the trunk returned as its [images, rows, columns, hidden_size] grid, which the
aligner groups.
DeepseekV41Vision.layout
Section titled “DeepseekV41Vision.layout”def layout() -> ViTLayoutDeepseekV41Vision.build
Section titled “DeepseekV41Vision.build”def build() -> nn.ModuleDeepseekV41Vision.geometry
Section titled “DeepseekV41Vision.geometry”def geometry() -> TowerGeometryGemma3nProjector
Section titled “Gemma3nProjector”class Gemma3nProjector(ProjectorBase)Gemma3nMultimodalEmbedder’s vision hard and soft token paths.
scales_soft: bool-
Whether soft features are multiplied by sqrt(
vision_width) before the embedder reads them: Gemma 3n’s image features are (modeling_gemma3n.py:1983 at transformers 5.17.0), and its audio embedder reads its encoder’s features as they are (:2192), which its norm’s epsilon tells apart.
Gemma3nProjector.setup
Section titled “Gemma3nProjector.setup”def setup()Gemma3nProjector.soft_embeddings
Section titled “Gemma3nProjector.soft_embeddings”def soft_embeddings(features)The reference embedder over soft features, without scales_soft’s scaling.
Gemma3nProjector.embed_hard
Section titled “Gemma3nProjector.embed_hard”def embed_hard(ids)Embed vocabulary ids in [vocab_offset, vocab_offset + vocab_size).
Pure: the host processor validates ids before device work.
Gemma3nProjector.merge_hard_embeddings
Section titled “Gemma3nProjector.merge_hard_embeddings”def merge_hard_embeddings(token_embeddings, ids)Replace this vocabulary range’s slots with hard embeddings, through the reference’s dummy id for every other slot.
Gemma3nVision
Section titled “Gemma3nVision”class Gemma3nVision( channel_multiplier: float = 1.0, stem_size: int = 64, stem_bias: bool = True, fix_stem: bool | None = None, in_chans: int = 3, pad_type: str = 'same', group_size: int | None = None, msfa_indices: tuple[int, ...] = (-2, -1), msfa_output_resolution: int = 16, layer_scale_init_value: float | None = 1e-05, drop_path_rate: float = 0.0,)Gemma 3n’s vision tower: MobileNet-v5’s encoder, by its construction
fields (MobileNetV5).
Gemma3nVision.build
Section titled “Gemma3nVision.build”def build() -> nn.ModuleGemma3nVision.geometry
Section titled “Gemma3nVision.geometry”def geometry() -> TowerGeometryGemma4Projector
Section titled “Gemma4Projector”class Gemma4Projector(ProjectorBase)Soft tokens to text width: a scale-free RMS norm, then the map.
The reference keeps this beside the tower (modeling_gemma4.py, Gemma4MultimodalEmbedder): the pre-projection norm carries no weight and the projection no bias.
Gemma4Projector.setup
Section titled “Gemma4Projector.setup”def setup()Gemma4Vision
Section titled “Gemma4Vision”class Gemma4Vision( hidden_size: int = 1152, intermediate_size: int = 4304, num_layers: int = 27, num_heads: int = 16, num_key_value_heads: int = 16, patch_size: int = 16, pooling_kernel_size: int = 3, position_embedding_size: int = 10240, hidden_act: str = 'gelu_pytorch_tanh', rms_norm_eps: float = 1e-06, rope_theta: float = 100.0, standardize: bool = False, use_clipped_linears: bool = False, head_dim: int | None = None,)A Gemma 4 trunk’s geometry, under the reference’s field names.
head_dim: int | None-
None retains hidden_size // num_heads; checkpoints may project a wider head.
Gemma4Vision.build
Section titled “Gemma4Vision.build”def build() -> nn.ModuleGemma4Vision.geometry
Section titled “Gemma4Vision.geometry”def geometry() -> TowerGeometryGemma4VisionTransformer
Section titled “Gemma4VisionTransformer”class Gemma4VisionTransformer( dtype: Dtype | None = None, precision: PrecisionLike = None,)The Gemma 4 vision trunk, param layout of Gemma4VisionConfig.
The processor supplies [B, patches, patch_pixels] and (x, y) position IDs with (-1, -1) for padding. Fixed NCHW images are patchified in the same HWC order. Outputs retain padded soft-token slots so their shape remains static under JIT; callers select valid features using processor lengths.
Each layer is the decoder block with RMS norms before and after both
halves (modeling_gemma4.py, Gemma4VisionEncoderLayer). Its attention has
bias-free maps over grouped key heads, the queries and keys normed with a
scale and the values without, its scores unscaled, and each spatial axis
turning its own half of the head the NeoX way (Gemma4VisionAttention,
apply_multidimensional_rope). The MLP gates the checkpoint’s GELU. With
use_clipped_linears every map clips its input and output to the bounds
the checkpoint stores (Gemma4ClippableLinear).
Gemma4VisionTransformer.setup
Section titled “Gemma4VisionTransformer.setup”def setup()GemmaProjector
Section titled “GemmaProjector”class GemmaProjector(ProjectorBase)Patch features into soft tokens: block average, norm, map to text width.
The reference reshapes the trunk sequence into its patch grid, averages each kernel block, norms and multiplies by its (vision, text) matrix (modeling_gemma3.py, Gemma3MultiModalProjector).
GemmaProjector.setup
Section titled “GemmaProjector.setup”def setup()Lfm3VlProjector
Section titled “Lfm3VlProjector”class Lfm3VlProjector(MergeProjector, ProjectorBase)LFM3-VL’s projector (Lfm3VlMultiModalProjector) is MergeProjector: a layer norm of each
trunk token at projector_layer_norm_eps, its bias kept whatever projector_bias says, each merge
block’s tokens joined unit by unit, and two maps, biased by projector_bias, around the
projector_hidden_act activation.
Lfm3VlVision
Section titled “Lfm3VlVision”class Lfm3VlVision( hidden_size: int = 1024, intermediate_size: int = 2560, num_hidden_layers: int = 14, num_attention_heads: int = 16, num_key_value_heads: int = 8, patch_size: int = 16, temporal_patch_size: int = 2, in_channels: int = 3, spatial_merge_size: int = 2, conv_kernel_size: int = 3, norm_eps: float = 1e-05, rope_theta: float = 1000000.0, layer_types: tuple[str, ...] | None = None,)LFM3-VL’s trunk geometry under its vision_config’s names.
Its layout (ViTLayout): a bias-free map of each patch’s channel-then-time pixels in merge-block
order (patchifier), an RMS norm after it (embedding_norm), and pre-norm blocks of RMS norms at
norm_eps with a bias-free SwiGLU feed-forward (w1 its gate, w3 its up map), each mixing its
frame’s patches by full attention, grouped and unbiased with an RMS norm on each head’s queries and
keys before the rotary turns them by the patch grid, rows then columns (full_attention), or by a
gated depthwise convolution over the frame’s grid (conv, GatedGridConv); no norm after the
blocks. Every RMS norm weighs its input after its cast back, as the release’s Lfm3VlRMSNorm does.
layer_types None is the release’s cadence over its depth (default_layer_types).
kinds: tuple[str, ...]-
Each layer’s kind:
layer_types, or the release’s cadence over the depth.
Lfm3VlVision.layout
Section titled “Lfm3VlVision.layout”def layout() -> ViTLayoutLfm3VlVision.build
Section titled “Lfm3VlVision.build”def build() -> nn.ModuleLfm3VlVision.geometry
Section titled “Lfm3VlVision.geometry”def geometry() -> TowerGeometryLlama4Projector
Section titled “Llama4Projector”class Llama4Projector(ProjectorBase)Tower output to text width: one bias-free map.
Llama4Projector.setup
Section titled “Llama4Projector.setup”def setup()Llama4Vision
Section titled “Llama4Vision”class Llama4Vision( hidden_size: int = 1408, intermediate_size: int = 5632, num_layers: int = 32, num_heads: int = 16, image_size: int = 336, patch_size: int = 14, num_channels: int = 3, layer_norm_eps: float = 1e-05, rope_theta: float = 10000.0, pixel_shuffle_ratio: float = 0.5, projector_input_dim: int = 4096, projector_output_dim: int = 4096,)A Llama 4 trunk’s geometry, under the reference’s field names.
Llama4Vision.layout
Section titled “Llama4Vision.layout”def layout() -> ViTLayoutLlama4Vision.build
Section titled “Llama4Vision.build”def build() -> nn.ModuleLlama4Vision.geometry
Section titled “Llama4Vision.geometry”def geometry() -> TowerGeometryLlama4VisionTransformer
Section titled “Llama4VisionTransformer”class Llama4VisionTransformer( dtype: Dtype | None = None, precision: PrecisionLike = None,)The Llama 4 vision trunk, param layout of Llama4VisionConfig.
pixel_values are what the checkpoint’s image processor emits: [B, C, H,
W] at image_size. The patches embed through one bias-free map, the class
token rides last through the trunk and is dropped before the adapter, and
what returns is the pixel-shuffled MLP output the outer projector maps to
text width.
Llama4VisionTransformer.setup
Section titled “Llama4VisionTransformer.setup”def setup()MergeProjector
Section titled “MergeProjector”class MergeProjector( norm: bool = True, norm_eps: float = 1e-06, norm_bias: bool | None = None, channel_major: bool = False, bias: bool = True, activation: str = 'gelu', activation_after: bool = False, post_norm: bool = False, dtype: Dtype | None = None, precision: PrecisionLike = None,)Each merge block’s features as one vector, mapped to the decoder width.
The trunk sequence comes in merge-block order (VisionTransformer). With norm, a layer norm
first at norm_eps (Qwen 3.5’s, MiMo’s and LFM3-VL’s); bias gives both maps their biases, and the
norm its own where norm_bias is unset (LFM3-VL’s norm keeps its bias either way). Each block’s
merge**2 features join unit by unit, or channel by channel where channel_major (Muse
Glimmer’s pixel_shuffle), and pass two maps with the checkpoint’s activation (its
hidden_act name) between them, and after the second too where activation_after (Muse
Glimmer’s adapter); with post_norm, an RMS norm at norm_eps after them (Kimi K3’s
PatchMergerMLPV2).
MiMoProjector
Section titled “MiMoProjector”class MiMoProjector(MergeProjector, ProjectorBase)MiMo’s merger (MiMoVisionPatchMerger): a layer norm, each merge block grouped, and two maps around an exact GELU to the decoder width. The release stores none of its biases.
MiMoVision
Section titled “MiMoVision”class MiMoVision( depth: int = 28, hidden_size: int = 1280, intermediate_size: int = 4608, num_heads: int = 32, num_key_value_heads: int = 8, qk_channels: int = 64, in_channels: int = 3, patch_size: int = 16, spatial_merge_size: int = 2, temporal_patch_size: int = 2, rms_norm_eps: float = 1e-06, fullatt_block_indexes: tuple[int, ...] = (0, 9, 18, 27), vit_window_attn_types: tuple[int, ...] = (-1, 0, 0, 0, 0, 1, 1, 1, 1) * 3 + (-1,), use_sink: bool = True, visual_token_window_size: int = 64,)MiMo ViT’s geometry under its vision_config’s names.
Its layout (ViTLayout): a bias-free patch map and no position table; RMS-normed blocks whose
grouped queries, keys and values come from one biased map, rotated by the patch grid as Qwen 3.5’s
are, then a biased SwiGLU. Each layer of fullatt_block_indexes attends over its whole frame,
every other one over the visual_token_window_size tokens either side of each query, plus a
learned logit per head on each frame’s first key where use_sink; a layer whose
vit_window_attn_types entry is 1 runs with the merge blocks taken column by column.
MiMoVision.layout
Section titled “MiMoVision.layout”def layout() -> ViTLayoutMiMoVision.build
Section titled “MiMoVision.build”def build() -> nn.ModuleMiMoVision.geometry
Section titled “MiMoVision.geometry”def geometry() -> TowerGeometryMuseGlimmerProjector
Section titled “MuseGlimmerProjector”class MuseGlimmerProjector(ProjectorBase)The merged image tokens as the decoder reads them (MuseGlimmerModel.get_image_features): each
merge block’s features channel by channel through the adapter’s two bias-free maps, each followed
by the activation (MergeProjector), a bias-free map to the decoder width, and a scale-free RMS
norm (perception_emb_norm).
MuseGlimmerVision
Section titled “MuseGlimmerVision”class MuseGlimmerVision( num_hidden_layers: int = 50, hidden_size: int = 1536, intermediate_size: int = 8960, num_attention_heads: int = 16, hidden_act: str = 'gelu', patch_size: int = 14, patch_temporal: int = 2, merge_size: int = 2, pos_emb_height: int = 32, pos_emb_width: int = 32, layer_norm_eps: float = 1e-05, rope_theta: float = 10000.0, layer_types: tuple[str, ...] = default_layer_types(50),)Muse Glimmer’s trunk geometry under its vision_config’s names.
Its layout (ViTLayout): a bias-free map of each raster patch’s time-then-channel pixels, the
learned pos_emb_height x pos_emb_width table resampled with half-pixel centres and zero
padding (MuseGlimmerVisionPatchEmbedder), a layer norm before the blocks and one after, and
CLIP-style blocks (layer norms at 1e-5, biased query, key and value maps, an ungated
feed-forward) rotated by the patch grid, columns then rows, each counted from one. The tokens run
window by window, pos_emb_height patches square; a window_attention layer attends within its
window, a full_attention one over its frame.
MuseGlimmerVision.layout
Section titled “MuseGlimmerVision.layout”def layout() -> ViTLayoutMuseGlimmerVision.build
Section titled “MuseGlimmerVision.build”def build() -> nn.ModuleMuseGlimmerVision.geometry
Section titled “MuseGlimmerVision.geometry”def geometry() -> TowerGeometryProjectorBase
Section titled “ProjectorBase”class ProjectorBase(checkpoint: ProjectorLayout | None = None)A native projector module whose constructor is its record schema. The registry names
its kind and checkpoint locates the tensors it consumes.
input_width: int | None-
Declared feature width when this projector consumes raw media without an encoder.
ProjectorLayout
Section titled “ProjectorLayout”class ProjectorLayout( prefix: str | tuple[str, ...], tensors: Mapping[str, tuple[str, ...]], strip: str = '', read: Callable[..., tuple[Variables, tuple[WeightLayout, ...]]] | None = None,)Where one projector kind’s tensors sit in a media checkpoint and the leaves they fill:
prefix before every name (or several, whose tensors each keep the name past their own),
tensors each name’s leaf path, strip a further prefix a name may carry (a merger’s), and
read the projector’s own reader where it checks its tensors (placed otherwise).
ProjectorLayout.path
Section titled “ProjectorLayout.path”def path(name: str) -> tuple[str, ...]The leaf of tensor name, shared by load and source export.
ProjectorLayout.variables
Section titled “ProjectorLayout.variables”def variables( hf_tensors: Mapping[str, Stored], param_dtype: str,) -> tuple[Variables, tuple[WeightLayout, ...]]The projector’s parameters, in the requested storage, and the layouts that write its tensors back from them.
Qwen35Projector
Section titled “Qwen35Projector”class Qwen35Projector(MergeProjector, ProjectorBase)Qwen 3.5’s projector fields: the trunk width, the merge size, the
merged width the text embeddings read directly. The merger
(modeling_qwen3_5.py, Qwen3_5VisionPatchMerger) is MergeProjector: norm,
each merge block grouped, two maps around an exact GELU.
Qwen35Vision
Section titled “Qwen35Vision”class Qwen35Vision( depth: int = 27, hidden_size: int = 1152, hidden_act: str = 'gelu_pytorch_tanh', intermediate_size: int = 4304, num_heads: int = 16, in_channels: int = 3, patch_size: int = 16, spatial_merge_size: int = 2, temporal_patch_size: int = 2, num_position_embeddings: int = 2304,)A Qwen 3.5 trunk’s geometry, under the reference’s field names.
Its layout (ViTLayout): a biased patch map, the learned table resampled with aligned corners,
CLIP’s layer at a 1e-6 epsilon (flax’s one-pass layer norm with its bias, the queries, keys and
values in one biased map, an ungated feed-forward) and full attention over each frame, rotated
by the patch grid, heights then widths (modeling_qwen3_5.py, Qwen3_5VisionRotaryEmbedding and
apply_rotary_pos_emb_vision). The trunk sequence leaves for the merger, its projector.
Qwen35Vision.layout
Section titled “Qwen35Vision.layout”def layout() -> ViTLayoutQwen35Vision.build
Section titled “Qwen35Vision.build”def build() -> nn.ModuleQwen35Vision.geometry
Section titled “Qwen35Vision.geometry”def geometry() -> TowerGeometrySiglipVision
Section titled “SiglipVision”class SiglipVision( hidden_size: int = 768, intermediate_size: int = 3072, num_layers: int = 12, num_heads: int = 12, image_size: int = 224, patch_size: int = 16, num_channels: int = 3, hidden_act: str = 'gelu_pytorch_tanh', layer_norm_eps: float = 1e-06,)A SigLIP trunk’s geometry, under the reference’s field names.
SiglipVision.layout
Section titled “SiglipVision.layout”def layout() -> ViTLayoutSiglipVision.build
Section titled “SiglipVision.build”def build() -> nn.ModuleSiglipVision.geometry
Section titled “SiglipVision.geometry”def geometry() -> TowerGeometryTowerBase
Section titled “TowerBase”class TowerBaseOne tower kind’s value: its fields, how it builds its module, and where its tensors sit in a
media checkpoint (checkpoint, None for a tower read another way, as the audio towers are).
TowerBase.build
Section titled “TowerBase.build”def build() -> nn.ModuleThe Flax module for this value.
TowerBase.geometry
Section titled “TowerBase.geometry”def geometry() -> TowerGeometryWhat an initialising input has to look like for this tower.
Nothing by default: a tower states the fields it has, and a conditioner reads them here instead of asking the value at runtime whether it carries each one.
TowerGeometry
Section titled “TowerGeometry”class TowerGeometry( image_size: int | None = None, patch_size: int | None = None, block_size: int | None = None, channels: int | None = None, mel_features: int | None = None, frames_per_token: int | None = None, position_size: int | None = None, retains_patch_slots: bool = False, pools_frames: bool = False,)The shapes a conditioner has to invent to create a tower’s media leaves.
A token-only init has no batch, so VisionConditioner and
AudioConditioner build one smallest input the tower accepts. What that
is differs by kind: a fixed-resolution tower states its image_size, a
patch-and-pool tower states the patch and block that make one, and an
audio tower states how many mel bins a frame carries. None means this
tower has no such field, and the conditioner uses its own default.
frames_per_token: int | None-
How many input frames an audio tower encodes into one token: its convolutions’ time strides times any frame reduction after them.
pools_frames: bool-
Whether an item’s frames pool into one frame’s tokens (MoonViT’s
tpool_patch_merger): its tokens are its first frame’s merge blocks, wherever it holds more.
TowerLayout
Section titled “TowerLayout”class TowerLayout( prefix: str, path: Callable[[str], tuple[str, ...] | None], read: Reader = placed, roots: tuple[str, ...] = (),)Where one tower kind’s tensors sit in a media checkpoint and the leaves they fill: prefix
before every name, and path from a name past it to its leaf, rooted at its collection
(params, and Gemma 4’s constants beside it). read binds them (placed, or, for a tower that
runs its patch convolution as one Dense map, translate_vit_weights). Loading
(tower_variables), routing a checkpoint’s names (hf_decoders) and binding an export
(interop.pretrained) read it alike.
roots: tuple[str, ...]-
The leading names of a tower stored under several roots, an empty
prefixbeside them, whose namespathreads whole: MiMo-V2.6’s audio tower, its encoder’s and its tokenizer folder’s.
TowerLayout.variables
Section titled “TowerLayout.variables”def variables( hf_tensors: Mapping[str, Stored], param_dtype: str,) -> tuple[Variables, tuple[WeightLayout, ...]]The tower’s variables from its tensors, in the requested storage, and the layouts that write each tensor it placed back from them; a tensor the tower does not compute with, as SigLIP’s pooling head, has none.
ViTLayout
Section titled “ViTLayout”class ViTLayout( depth: int, hidden_size: int, num_heads: int, num_kv_heads: int, head_dim: int, intermediate_size: int, activation: str | Callable[[jax.Array], jax.Array], in_channels: int, patch_size: int, temporal_patch_size: int, spatial_merge_size: int, norm_type: Literal['rms', 'layer', 'fast_layer'], norm_eps: float | None, norm_bias: bool, mlp_bias: bool, packed: bool, patch_bias: bool, attention: tuple[Attention, ...], order: tuple[Order, ...], sinks: tuple[bool, ...], positions: LearnedPositions | None = None, special_tokens: SpecialTokens = SpecialTokens(), patch_projection: Literal['dense', 'conv'] = 'dense', image_size: int | None = None, rotary: Literal['half', 'adjacent'] | None = 'half', pre_norm: Literal['layer', 'rms'] | None = None, post_norm: Literal['layer', 'rms'] | None = None, post_norm_scope: Literal['sequence', 'class'] = 'sequence', post_norm_affine: bool = True, outer_norm_eps: float | None = 1e-05, mlp_packed: bool = False, output: Literal['sequence', 'patches', 'patch_grid'] = 'sequence', frame_positions: int | None = None, pool_frames: bool = False, branch_scale: float | None = None, rope_theta: float = 10000.0, rotary_columns_first: bool = False, rotary_interleaved: bool = False, rotary_origin: int = 0, band: int = 0, window: int = 0, merge_blocks: bool = True, channels_first: bool = True, attention_bias: bool = True, qk_norm: bool = False, conv_kernel: int = 3, scale_after_cast: bool = False,)The geometry and wiring of one family’s tower.
attention, order and sinks name each layer’s: its keys (a frame’s every patch, a row’s every
patch across its frames (item, MoonViT’s), band keys
either side in the layer’s order, or the patches of its window-patch square) or, for conv, a gated
conv_kernel-square convolution over its frame’s patch grid in the attention’s place (LFM3-VL’s,
GatedGridConv, which reads each patch’s grid place as its rotary position), the order its tokens run
in (the processor’s, merge blocks column by column, or window by window), and whether a learned logit
per head joins the first key of each frame (MiMo’s sinks). rotary_columns_first turns the first
rotary half by the column and the second by the row, both counted from rotary_origin, or under
rotary_interleaved the two axes pair by pair (MoonViT’s); positions keeps a fixed grid or resizes
its square table by its stated interpolation. Bilinear sampling without aligned corners gives out-of-
range taps zero weight (F.grid_sample’s zero padding). merge_blocks is the processor’s patch
order: merge block by merge block, else raster; channels_first its pixel order in a patch, channels
then time, else time then channels. mlp_packed keeps a gated feed-forward’s gate and up maps in one
(as packed the attention’s queries, keys and values). post_norm names the norm after the blocks, a
layer or an RMS norm, or none. output selects the sequence, patches alone or a [B, rows, columns, F] patch grid; post_norm_scope distinguishes CLIP’s normalized class row from a normalized whole
sequence. pre_norm names the norm after the patch map, as post_norm does. attention_bias biases
the attention’s maps, and qk_norm norms each head’s queries and keys (an RMS norm at the layers’
epsilon) before the rotary; under scale_after_cast every RMS norm weighs its normalized input after
its cast back, as Llama’s does. An RMS norm’s epsilon None (norm_eps, outer_norm_eps) is the
machine epsilon of the dtype its input accumulates in (RMSNorm.epsilon), as torch’s
RMSNorm(eps=None) takes it (Kimi K3’s MoonViT).
frame_positions: int | None-
A fixed 1-D sinusoidal table of this many frames, MAE’s computed in float32 (
dew.nn.rope.sinusoid_frequencies’powered), added with the learned positions to each frame’s patches of a row of more than one frame (MoonViT’sLearnable2DInterpPosEmbDivided_fixed). pool_frames: bool-
Whether a row’s merge blocks are averaged over its frames after the blocks, the row returning its first frame’s places and zeros past them (MoonViT’s temporal pooling,
tpool_patch_merger).
VisionTransformer
Section titled “VisionTransformer”class VisionTransformer(dtype: Dtype | None = None, precision: PrecisionLike = None)A family’s trunk (ViTLayout): patches mapped and placed, the blocks, and the trunk sequence
in merge-block order.
Packed pixels [B, patches, pixels] come with each row’s [time, height, width] patch grid, a
row’s padding past its patches; attention stays inside a frame, or a window of it, and padding
(segment 0) sees nothing. NCHW images are one frame each, repeated along time, packed in the
processor’s order. Each layer runs in its own token order, the tokens, their rotary places and
their frames gathered into it from the previous layer’s; the sequence returns to the processor’s
order after the last.
VisionTransformer.setup
Section titled “VisionTransformer.setup”def setup()VisionTransformer.image_lengths
Section titled “VisionTransformer.image_lengths”def image_lengths(sample_shape: tuple[int, ...], rows: int) -> np.ndarrayVisionTransformer.model_flops
Section titled “VisionTransformer.model_flops”def model_flops(lengths) -> floatVisionTransformer.sequence_and_pool
Section titled “VisionTransformer.sequence_and_pool”def sequence_and_pool(pixel_values, grid_thw=None) -> tuple[jax.Array, jax.Array]The selected sequence and its class row through the declared post norm.
projector_layout
Section titled “projector_layout”def projector_layout(kind: str) -> ProjectorLayoutWhere projector kind kind’s tensors sit: its class’s checkpoint
(dew.registry.projectors).
projector_variables
Section titled “projector_variables”def projector_variables( kind: str, hf_tensors: Mapping[str, Stored], param_dtype: str,) -> tuple[Variables, tuple[WeightLayout, ...]]One projector kind’s parameters, in the requested storage (projector_layout), and the
layouts that write its tensors back from them.
tower_layout
Section titled “tower_layout”def tower_layout(kind: str) -> TowerLayoutWhere tower kind kind’s tensors sit: its class’s checkpoint (dew.registry.towers).
tower_variables
Section titled “tower_variables”def tower_variables( kind: str, hf_tensors: Mapping[str, Stored], param_dtype: str,) -> tuple[Variables, tuple[WeightLayout, ...]]One vision tower’s variables, in the requested storage (tower_layout), and the layouts
that write its tensors back from them.