Skip to content

dew.inputs.condition

What every condition encoder is (ConditionEncoder) and the work it does (ConditionWork), apart from the encoders themselves (dew.inputs.encoders), so a spec of a model’s inputs, a protocol or a model reads the contract without the towers.

ConditionEncoder.bound(record) rebuilds an encoder from a run’s record, the registry’s {"class": ..., "fields": ...} whose fields recorded_fields wrote.

NameSummary
RawOne item of a modality’s raw data: a prompt, a waveform.
EncodedThe conditioning value a modality’s encoder produces: a TextContext, a DenoisingCondition, an array.
ConditionWorkA condition encoder’s forward FLOPs and the sequence its denoiser reads.
ConditionEncoderTurns one modality’s raw data into a conditioning value.

attribute source

Raw = TypeVar('Raw')

One item of a modality’s raw data: a prompt, a waveform.

attribute source

Encoded = TypeVar('Encoded', default=object)

The conditioning value a modality’s encoder produces: a TextContext, a DenoisingCondition, an array. An encoder that leaves it unstated promises its callers nothing about the value beyond what it hands the model.

dataclass source

class ConditionWork(
flops: float,
lengths: np.ndarray,
spans: np.ndarray,
features: int,
pooled_features: int | None = None,
time_ids: int | None = None,
)

A condition encoder’s forward FLOPs and the sequence its denoiser reads.

The output lengths may differ from encoded input (system prompts cropped, CLIP and T5 concatenated, or visible zero tokens retained). Keeping both under the encoder makes its geometry the encode contract.

def of_tokens(tower: Counted, tokens: Mapping[str, np.ndarray], features: int) -> Self

The work of a tower that reads each row’s unmasked tokens and hands its denoiser the whole fixed-length sequence, features wide: its FLOPs over the real tokens, their lengths, and spans of the token budget.

class source

class ConditionEncoder(ABC, Generic[Raw, Encoded])

Turns one modality’s raw data into a conditioning value.

Subclasses implement from_pretrained, tokenize, encode and recorded_fields, the record a run keeps of them (dew.registry.Recorded).

parameter_collections: tuple[str, ...] | None

The collections that hold learned weights, frozen ones included, or None for a bare parameter tree. Collections not named here keep their dtype.

reads_captions: bool

Whether a dataset’s captions are this encoder’s raw data. An encoder that does not read captions reads a batch field its dataset writes itself, such as audio, and InputSpec.tokenize leaves that field alone.

keyword: str

The model keyword this encoder’s value is passed under. Each value type goes to the models written for it: a TextContext to the native architectures’ textcontext, and a DenoisingCondition to the published families’ conditioning.

def parameter_axes() -> dict[tuple[str, ...], tuple[str | None, ...]]

The axes each parameter of params is drawn in, by its path in params, read off its towers’ draws (drawn, dew.nn.sharding.boxed_axes): the table that places a run holding this encoder frozen (dew.objectives.diffusion.DiffusionObjective.parameter_axes) and a pipeline loaded on a mesh. An encoder whose towers name no axes holds none.

def drawn() -> Mapping

Its towers’ parameters as their draws make them, traced for shapes alone (jax.eval_shape, no array made), laid out as params holds them: the shapes a pipeline loaded without them draws its denoiser over (dew.interop.pretrained.pipeline_axes). None drawn for an encoder that draws none.

def from_pretrained(checkpoint: str, *, params: Variables | None = None) -> Self

Load the tower named checkpoint. This is the one call that opens files.

Anything else a checkpoint needs is a keyword field with a default; recorded_fields records those fields, and bound rebuilds the encoder from them. When you pass params, the load uses them as they are. It reads the checkpoint’s metadata but none of its weights, and keeps the values, dtypes and placement of params.

def tokenize(texts: Sequence[Raw]) -> Mapping[str, np.ndarray]

Turn raw data into the host arrays encode reads, one row per item.

def encode(params: Variables, tokens) -> Encoded

Encode tokens into the conditioning value on device, under params.

def captions(tokens) -> tuple[str, ...]

Return the text the tokens stand for, to show in a rendered artifact.

A modality that is not text returns an empty tuple.

def recorded_fields() -> Mapping[str, object]

Return the keyword fields that from_pretrained rebuilds this encoder from, its checkpoint among them: the record a run keeps (dew.registry.Recorded).

def dependencies() -> Mapping[str, object]

What tokenize makes of raw data depends on: the encoder and the fields it is rebuilt from, which hold its context length and its checkpoint’s revision.

def bound(
record: Mapping[str, object],
*,
params: Variables | None = None,
) -> ConditionEncoder

Rebuild the encoder a record names, {"class": alias or import path, "fields": ...} (dew.registry.component), around params where given.

The fields are the keywords recorded_fields wrote, scalars and dtype names its from_pretrained reads itself, passed back to it, so each encoder keeps its own concrete signature. checkpoint is the one field every encoder takes; a record without one raises ValueError. With params the load reads the checkpoint’s metadata and tokenizer, never its weights (from_pretrained).