Skip to content

dew.inference

Inference tasks: native generation with a model and its weights, and clients for external engines, each imported only when you use its API.

So a text-to-image task does not import the run loader’s checkpointing, the data pipeline or the rollout servers.

NameSummary
BlockGenerationGenerates block-diffusion canvases from a block denoiser and its weights. Documented in dew.inference.tasks.
CheckpointBanksServes banks by restoring a run’s published weights one bank at a time. Documented in dew.inference.banks.
CompletionThe choices of one completion request in prompt-major order, with the SDK responses kept.
DenoisingInputsEncoded conditioning and initial noise, placed on the devices the way a call uses them.
DrawOne model call as the engine served it: the one record a server, an environment, a session and the packer read.
EngineServes kinds from one thread of its own between start and close: each turn, every busy kind takes one step, so a kind’s long queue never starves another’s; with none busy the thread waits for a submission.
HeldBanksServes banks from a variables tree already held in memory. Documented in dew.inference.banks.
ImageRequestOne request a served text-to-image task answers (TextToImage.scoring): its prompts, an image each, the key its draws come from, and the settings of its lone call task(prompts, key=key, ...): steps, guidance and solver (the scoring task’s where omitted), and negative, its negative prompts, one or one a prompt, in place of the task’s blank.
ImagesDecoded samples in [-1, 1], NHWC (NTHWC for a video field), placed as the task ran them.
InfrastructureErrorA rollout’s infrastructure failed: a connection, an engine, a sandbox, a runtime or a service.
KindOne serving kind, as Engine drives it.
LayerBanksServes a banked store’s values, read one bank at a time. Documented in dew.inference.banks.
MaskedGenerationSamples a response with native MDLM, holding the prompt fixed: the whole response at once (Unmask), or a block of block places at a time as LLaDA2’s releases walk (DiscreteProcess.generate’s block, Confident’s reveal). Documented in dew.inference.tasks.
NCCLPushPublishes a policy version to vLLM replicas by NCCL broadcast from a trainer GPU.
NativeRolloutServerServes rollouts from Dew’s Server, stepped by an Engine of its own, or where the server’s mesh spans several processes (pooled) by the run itself, every process together (waited).
OllamaCompletionCalls one Ollama model through an ollama.Client or ollama.AsyncClient that you pass in.
OpenAICompletionCalls one model through an OpenAI client, including one pointed at a vLLM or SGLang base_url.
OpenAIRolloutServerServes rollouts from the OpenAI-compatible completions endpoint of a vLLM or SGLang engine.
ProcessorInterface for a loaded source’s host processor, which prepares requests and decodes tokens. Documented in dew.inference.tasks.
PublicationPublishes weights to an engine fleet under a version, for use as a scheduler’s Publisher.
RolloutServerSamples continuations of token prompts under a versioned policy whose weights can be reloaded.
RunProcessorAdapts a run’s tokenizer to the Processor interface a task uses.
SafetensorsBanksReads a local HF decoder checkpoint without materializing its layer stack. Documented in dew.inference.banks.
SafetensorsReloadPublishes a policy version to engine replicas through safetensors on disk.
ScoringA task served by forward passes over its requests’ rows (ScoringServer).
ScoringRequestA request as its task admitted it: its rows, each scored in one of the task’s passes, and what the task’s answer reads of it besides their outputs.
ScoringServerAnswers scoring requests through one task’s passes, one pass a step.
ServerServes text requests with continuous batching over one resident KV cache.
TextGenerationGenerates next tokens from a decoder, its weights and its processor. Documented in dew.inference.tasks.
TextToImageGenerates images from text prompts with a trained diffusion model and its encoders.
TimedA request’s future, with when it was submitted, admitted to a step and finished, in time.perf_counter seconds; admitted and finished are None until they happen.
UsageToken usage the backend reported for the whole request.
VLLMGenerateServerServes rollouts from vLLM’s token route, POST /inference/v1/generate.
pipelineLoad the inference task for source, with its weights placed once.

dataclass source

class Completion(
texts: tuple[str, ...],
finish_reasons: tuple[str | None, ...],
token_counts: tuple[int | None, ...],
usage: Usage | None,
responses: tuple[OllamaResponse | OpenAIResponse, ...],
tokens: tuple[tuple[int, ...] | None, ...],
log_probs: tuple[tuple[float, ...] | None, ...],
routed_experts: tuple[np.ndarray | None, ...] = (),
)

The choices of one completion request in prompt-major order, with the SDK responses kept.

token_counts holds per-choice counts, which stay None when only aggregate usage was reported. finish_reasons keeps the backend’s values, including an absent reason. tokens and log_probs hold each choice’s sampled ids and their log-probabilities where the backend reported them; a vLLM completion asked for logprobs with return_tokens_as_token_ids in extra_body reports both. The log-probabilities come from whatever distribution the engine was configured to report, and this record does not relabel them. routed_experts holds each choice’s [forwarded ids, layers, top_k] expert record when vLLM ran with --enable-return-routed-experts.

dataclass source

class DenoisingInputs(
conditions: Mapping[str, Conditioning] = struct.field(default_factory=dict),
unconditional: Mapping[str, Conditioning] = struct.field(default_factory=dict),
rows: int | None = struct.field(pytree_node=False, default=None),
grid_steps: int | None = struct.field(pytree_node=False, default=None),
process: Process | None = struct.field(pytree_node=False, default=None),
times: tuple[float, ...] | None = struct.field(pytree_node=False, default=None),
)

Encoded conditioning and initial noise, placed on the devices the way a call uses them.

TextToImage.prepare builds them, and a TextToImage call takes them in place of prompts. rows is the number of this process’s real prompts. On a mesh the arrays hold the padded batch, sharded by row, which a call uses as it is. A call refuses inputs prepared for another step count, another geometry or another mesh.

dataclass source

class Draw(
prompt: tuple[int, ...],
tokens: tuple[int, ...],
behavior_log_probs: tuple[float, ...],
finish_reason: str,
version: int,
raw_log_probs: tuple[float, ...] | None = None,
routed_experts: Routing | None = None,
support: tuple[tuple[int, ...], ...] | None = None,
)

One model call as the engine served it: the one record a server, an environment, a session and the packer read.

prompt are the ids the engine read and tokens the ids it sampled, with the id that ended the draw included when it stopped: an EOS id, or the id completing a stop string (check_ending). behavior_log_probs is each sampled id’s likelihood under the distribution that drew it, and raw_log_probs the unmodified model’s, or None where the backend cannot report it (vLLM, SGLang, a recording gateway). finish_reason is one of FINISH_REASONS, and the draw terminated where it is stop or tool_calls. version is the policy version being served when the request was submitted, the oldest policy that may have produced any of its ids.

Two engine records are optional. routed_experts is the mixture’s routing for every id the engine forwarded, as [len(prompt) + len(tokens) - 1, layers, top_k] expert ids in the dtype the engine sent; the last sampled id is never forwarded. vLLM calls this record routed_experts and SGLang meta_info.routed_experts, and the trainer replays it (dew.nn.protocols.Routes). A model whose layers route at several sites, a routed mixer beside a routed feed-forward, records one such array a site, keyed by its path (Routing). support holds, for each sampled id, the token ids the sampler’s top-k/top-p filters kept (vLLM’s sampling_mask). With a support, the behavior log-probability is the filtered one, and the trainer renormalizes over the same support. None means no filter. Ids, likelihoods and records that do not match the sampled ids one to one, or are not finite, raise ValueError.

terminated: bool

Whether the draw ended itself, on an EOS id, a stop string or a tool call, rather than being cut at its budget or aborted.

def checked(sampling: Sampling) -> Draw

Admit a nonempty sampled action ending as sampling ends one (check_ending). Imported call records may be empty, but an action presented by a sampling server may not.

def response(sampling: Sampling) -> tuple[int, ...]

The drawn ids a reward reads decoded (response).

class source

class Engine(*kinds: Kind = (), name: str = 'serving-engine')

Serves kinds from one thread of its own between start and close: each turn, every busy kind takes one step, so a kind’s long queue never starves another’s; with none busy the thread waits for a submission. A step that fails stops every kind, each request settled with the failure, as one device’s failure is every kind’s.

A request’s callback runs on the serving thread, inside its kind’s step, and may reload or close from there. A program that ends without close stops the thread after the step in progress (dew._coordination.stop_at_exit).

def start() -> None
def close() -> None

Stop serving once what waits is answered.

def stop() -> None

Stop serving after the step in progress; what waits is left to its kind’s owner.

def reload(reloading: Callable[[], None]) -> None

Run reloading, a kind’s reload, between two steps.

dataclass source

class ImageRequest(
prompts: str | Sequence[str],
key: int | jax.Array,
steps: int | None = None,
guidance: Guidance | float | None | Omitted = OMITTED,
solver: Solver | None = None,
negative: str | Sequence[str] | None = None,
prepare_key: int | jax.Array | None = None,
prepare_steps: int | None = None,
on_step: Callable[[int, np.ndarray], None] | None = None,
)

One request a served text-to-image task answers (TextToImage.scoring): its prompts, an image each, the key its draws come from, and the settings of its lone call task(prompts, key=key, ...): steps, guidance and solver (the scoring task’s where omitted), and negative, its negative prompts, one or one a prompt, in place of the task’s blank. A request whose noise and conditions are prepared apart from its walk, as task(task.prepare(prompts, key=prepare_key, steps=prepare_steps, unconditional=negative), key=key, steps=steps, ...) prepares them, names prepare_key and prepare_steps (its key and steps where None). on_step, where given, streams the walk: it is called on the runtime’s callback thread with each solver step’s number, from 1, and the clean prediction that step reads, the request’s rows as a host array, as the walk runs.

dataclass source

class Images(
rows: int | None = struct.field(pytree_node=False, default=None),
latents: ArrayT | None = None,
)

Decoded samples in [-1, 1], NHWC (NTHWC for a video field), placed as the task ran them.

latents holds the denoised latents, and images is None when the call ran with decode=False. host() copies this process’s rows real rows back to host arrays, and pil() returns an image batch’s real rows as 8-bit images.

def host() -> Images[np.ndarray]

Return this process’s real rows as host arrays, without the padding added to fill the devices.

def pil() -> list[PILImage]

Return this process’s real rows of an NHWC image batch as PIL images.

Three channels give RGB images and one channel gives grayscale. The pixels are quantized by dew.artifacts.uint8_pixels. Raises ValueError when the samples kept only their latents, or when they are not [N, H, W, 3] or [N, H, W, 1] images.

class source

class InfrastructureError(RuntimeError)

A rollout’s infrastructure failed: a connection, an engine, a sandbox, a runtime or a service.

A session source turns it, like any OSError, into an INFRA_ERROR session, which the scheduler retries; any other error is broken code or a broken contract and fails the rollout. The remote rollout servers raise it when the engine cannot be reached, times out, or answers 5xx or 429, and Dew’s sandbox runners when they cannot run a program. An environment, verifier or reward raises it for a failure of its own external boundary that is not an OSError, such as an httpx transport error.

class source

class Kind(Protocol)

One serving kind, as Engine drives it.

arrived: Callable[[], None]

Called after each submission, so an engine waiting for work wakes to it.

pooled: bool

Whether its steps span several processes, which step it together (dew.inference.Server.run); an engine steps a kind of one process, as only that process’s submissions wake it.

busy: bool

Whether a request waits for, or runs in, a step.

def step() -> None

One bounded device call: admit what fits, run it, settle what finished.

def fail(failure: BaseException) -> None

Stop after failure: every waiting and running request settles with it.

dataclass source

class NCCLPush(
source: Pretrained,
engines: tuple[str, ...],
library: str | Callable[[int], Collectives],
dtype: str = 'bfloat16',
chunk: int = 512 * 2 ** 20,
timeout: float = 600.0,
)

Publishes a policy version to vLLM replicas by NCCL broadcast from a trainer GPU.

The replicas run vLLM (checked against v0.30.0) launched with --weight-transfer-config '{"backend": "nccl"}' and VLLM_SERVER_DEV_MODE=1. The module docstring of dew.inference.nccl describes the handshake and the push in full.

  • source is the Pretrained the replicas were launched from; its export gives the tensors, of the trained tree and of every other tree the source exports (a decision release’s readout), each read home by the pool as process 0 sends it (reading_home).
  • engines are the replicas’ root URLs, not their /v1 API URLs.
  • library is the engine’s own libnccl.so.2, because NCCL refuses a peer of another version when the group opens; or a maker of the push’s Collectives from the sending device’s CUDA ordinal, which makes its calls instead (an emulation of NCCL over the host).
  • chunk bounds the bytes of tensors placed on the sender device beside the trainer’s state. The first chunk is read before any replica pauses, and each next one is read and copied over on a reader thread while the current one broadcasts, so a push holds at most two chunks there, or two of its largest tensor.

Floating-point leaves are cast to dtype before the push, and timeout bounds each HTTP call in seconds. Groups open on the first push and stay open until close. Every process of a pool calls the push.

timings holds the seconds each phase of the last push took: export (the publication’s specs and chunks, before any tensor is read, on every process), and on process 0, which sends, send (reads, copies and broadcasts until every replica has loaded the tensors) and total.

def close() -> None

Abort the NCCL groups this side opened. A later push opens them again.

Groups are aborted rather than destroyed, because NCCL’s destroy finalizes the group, which waits on the engine’s side, and an engine keeps its side open until it exits. A failed push also calls close, because a group whose broadcast or replica failed cannot deliver the next version.

class source

class NativeRolloutServer(server: Server, *, version: int = 0)

Serves rollouts from Dew’s Server, stepped by an Engine of its own, or where the server’s mesh spans several processes (pooled) by the run itself, every process together (waited).

The engine orders weight loads with the server’s steps, so a load happens between two steps. A submission waits for a load in progress but not for a step. The server takes it while a step runs and seats it at the next one. The server’s own sampling policy and transforms decide the draws, and a draw keeps both the raw and the behavior likelihoods the server records. The engine’s thread starts at the first submission, so constructing the server starts nothing. If the program ends without close, the engine stops after the step in progress.

A draw ends on one of the policy’s EOS ids or on the id completing one of its stop strings, and Draw.terminated tells those from a draw cut at its budget or by a criterion that truncates it (MaxNewTokens). A server whose task ends rows on a criterion of its own besides those, which a draw’s policy cannot name, raises ValueError.

deferred builds one whose server is built from the first version it is loaded with, as a run that builds its rollout before its weights are drawn serves them (RolloutScheduler.served).

engine: Engine | None

The engine that steps the server, and orders a load with its steps; None for a deferred server not yet loaded, and for a pooled one, which the run steps (waited).

pooled: bool

Whether the server’s mesh spans several processes, which step it together (waited).

def deferred(
build: Callable[[Variables, int], Server],
sampling: Sampling,
) -> NativeRolloutServer

A server that draws nothing until its first load, which builds its Server with build from the loaded variables and version, the policy drawing under sampling: a run’s rollout built before the run draws its weights. Its version before then is -1, behind every update.

def submit(
prompt: Sequence[int],
max_new_tokens: int,
*,
key: int | jax.Array,
) -> Future[Draw]

Queue one draw and return its future.

A request the server refuses raises here, and the other requests keep running.

def load(variables: Variables, version: int) -> None
def waited(pending: Callable[[], Sequence[Future]]) -> None

Return once pending, read afresh as draws finish, gives no future: as each finishes where an engine steps the server, and for a pooled server stepped with every process of its mesh, on the calling thread, until no process waits on one. Every process calls it at the same point of its run with the futures it waits on, none for a process that waits on nothing. Each round agrees whether any process’s server holds work, which every process then steps (Server.step), and else waits for its own futures’ work beside the server (a reward scored, an environment stepped) to finish one or to submit a draw. The steps run where the run’s other programs run, in the same order on every process: a thread of their own would enter the mesh’s collectives in another order on each, and its agreements interleave with the run’s (ledger 4279). A step that fails on any process stops the server on every one, each step’s outcome agreed before the next round (agreed), every request it holds settled with the failure (Server.fail).

def close() -> None

Stop after the step in progress; every draw still outstanding fails.

dataclass source

class OllamaCompletion(model: str, client: OllamaClient | AsyncOllamaClient)

Calls one Ollama model through an ollama.Client or ollama.AsyncClient that you pass in.

Calling it, or acall with the async client, takes a batch of text prompts and a finite token budget and returns a Completion. Each prompt is one generate request, seeded at the root seed key plus its row index; pass the seed as key=, not seed=. Other request fields follow the SDK schema, including images, format, context, raw, template, system, think and logprobs. options takes the SDK’s Options value or a mapping of backend options. chat and achat support tools and tool results, and stream and astream yield the SDK’s own GenerateResponse objects.

def stream(
prompt: str,
max_new_tokens: int,
*,
key: int | jax.Array | None = None,
**parameters: RequestField = {},
) -> Iterator[OllamaResponse]
def chat(
messages: Sequence[ChatMessage],
max_new_tokens: int,
*,
key: int | jax.Array | None = None,
stream: bool = False,
**parameters: RequestField = {},
) -> OllamaChat | Iterator[OllamaChat]
async def acall(
prompts: str | Sequence[str],
max_new_tokens: int,
*,
key: int | jax.Array | None = None,
**parameters: RequestField = {},
) -> Completion
async def astream(
prompt: str,
max_new_tokens: int,
*,
key: int | jax.Array | None = None,
**parameters: RequestField = {},
) -> AsyncIterator[OllamaResponse]
async def achat(
messages: Sequence[ChatMessage],
max_new_tokens: int,
*,
key: int | jax.Array | None = None,
stream: bool = False,
**parameters: RequestField = {},
) -> OllamaChat | AsyncIterator[OllamaChat]

dataclass source

class OpenAICompletion(
model: str,
client: OpenAI | AsyncOpenAI,
provider: Literal['openai', 'vllm', 'sglang'] = 'openai',
)

Calls one model through an OpenAI client, including one pointed at a vLLM or SGLang base_url.

provider (openai, vllm or sglang) names the engine behind the client. The SDK’s own request options pass through to the completions and chat resources. Engine-only controls that vLLM and SGLang both accept, such as top_k, min_p and stop_token_ids, go in extra_body. A Sampling passed as sampling= becomes request fields, and its top-k, min-p, repetition-penalty and EOS controls need provider="vllm" or "sglang". SDK responses and streaming chunks keep the backend’s logprobs, token ids and extensions, tool calls and structured output fields unchanged. Completion prompts are text or rows of token ids, and a row of ids reaches the engine as ids, without being detokenized and tokenized again.

def stream(
prompts: str | Sequence[str] | TokenRows,
max_new_tokens: int,
*,
key: int | jax.Array | None = None,
**parameters: RequestField = {},
) -> Stream[OpenAIResponse]
def chat(
messages: Sequence[ChatMessage],
max_new_tokens: int,
*,
key: int | jax.Array | None = None,
stream: bool = False,
**parameters: RequestField = {},
) -> ChatCompletion | Stream[ChatCompletionChunk]
async def acall(
prompts: str | Sequence[str] | TokenRows,
max_new_tokens: int,
*,
key: int | jax.Array | None = None,
**parameters: RequestField = {},
) -> Completion
async def astream(
prompts: str | Sequence[str] | TokenRows,
max_new_tokens: int,
*,
key: int | jax.Array | None = None,
**parameters: RequestField = {},
) -> AsyncStream[OpenAIResponse]
async def achat(
messages: Sequence[ChatMessage],
max_new_tokens: int,
*,
key: int | jax.Array | None = None,
stream: bool = False,
**parameters: RequestField = {},
) -> ChatCompletion | AsyncStream[ChatCompletionChunk]

class source

class OpenAIRolloutServer(
completion: OpenAICompletion,
sampling: Sampling,
weights: WeightSync,
*,
version: int = 0,
workers: int = 64,
processed_logprobs: bool = False,
routing: bool = False,
decode: Callable[[Sequence[int]], str] | None = None,
)

Serves rollouts from the OpenAI-compatible completions endpoint of a vLLM or SGLang engine.

Each submission is one completion request with token ids as the prompt. It includes the Sampling policy and a seed, and asks for one log-probability per sampled token and for the sampled ids themselves. Up to workers requests are in flight at once. The engine is completion.provider, which must be vllm or sglang.

The reported log-probabilities are behavior likelihoods only for some policies. vLLM reports raw ones unless started with --logprobs-mode processed_logprobs; pass processed_logprobs=True when it is. vLLM returns a filtering policy’s kept ids only on its token route, so use VLLMGenerateServer for a filtering policy. SGLang’s /v1/completions reports the temperature-scaled distribution before its filters, so with SGLang any temperature works but no filter does, and SGLANG_RETURN_ORIGINAL_LOGPROB must stay unset. An SGLang draw streams, drawn as the policy without its EOS ids and cut at the first of them (_streamed). Both engines end a draw on the policy’s stop strings themselves, matched on the text they decode, and return its ids through the one completing the string, but vLLM keeps a speculative block’s ids past it (vllm/v1/engine/detokenizer.py:118-131 at v0.30.0). So a policy with stop strings takes decode, the policy’s tokenizer’s, and its draws are cut at the first id whose text completes one (through_stop). The constructor raises ValueError for the combinations these rules exclude, and for a temperature of zero. SGLang honors a seed only under --enable-deterministic-inference.

With routing=True, every draw records vLLM’s routed experts for routing replay (dew.nn.protocols.Routes); the engine must run with --enable-return-routed-experts.

class source

class Publication(
weights: WeightSync,
*,
version: int = 0,
stamp: Callable[[int], None] | None = None,
)

Publishes weights to an engine fleet under a version, for use as a scheduler’s Publisher.

load(variables, version) pushes the weights, stamps the version, then updates version. weights does the push (SafetensorsReload, or any WeightSync). stamp, when set, labels later calls with a version, as a recording gateway does (dew.interop.gateway.Gateway.stamp). The stamp runs only after every replica serves the new version, because a gateway stamps a call when it arrives, and a stamp ahead of a replica would claim weights the call was not sampled from. A failed push or stamp raises and leaves version unchanged. The launch version is stamped when the version is first read, which a scheduler does before it submits a draw under it, so a gateway left at a higher version by an earlier run cannot mislabel this run’s first calls, and constructing the publication touches no gateway. Every process calls load, and process 0 stamps.

def load(variables: Variables, version: int) -> None

class source

class RolloutServer(Protocol)

Samples continuations of token prompts under a versioned policy whose weights can be reloaded.

submit returns a future without waiting for generation. load replaces the served weights and sets version, and draws submitted after it report that version.

def submit(
prompt: Sequence[int],
max_new_tokens: int,
*,
key: int | jax.Array,
) -> Future[Draw]
def load(variables: Variables, version: int) -> None
def close() -> None

dataclass source

class RunProcessor(tokenizer: Tokenizer)

Adapts a run’s tokenizer to the Processor interface a task uses.

Calling it encodes each text prompt and left-pads the rows into ModelInputs; a text run takes no images or audio. decode returns one string per token row.

bos_id: int | None

The id the run’s tokenizer starts a sequence with, or None.

def decode(tokens: jax.typing.ArrayLike) -> list[str]
def encode(text: str, *, specials: bool) -> list[int]

text’s ids from the run’s tokenizer, with the special tokens it adds around a sequence where specials, and none otherwise.

def spelling(ids: Sequence[int]) -> str

The text the run’s tokenizer decodes ids to, its special tokens spelled out.

dataclass source

class SafetensorsReload(
source: Pretrained,
directory: Path,
engines: tuple[str, ...],
engine: Literal['vllm', 'sglang'],
dtype: str = 'bfloat16',
timeout: float = 600.0,
)

Publishes a policy version to engine replicas through safetensors on disk.

source.save writes the weights, the derived config and the tokenizer files into directory, the directory every replica was launched on (a shared one when replicas span hosts). Floating-point leaves are cast to dtype first. The source’s writers replace each file whole (dew._files.replacing) and publish a sharded index last, so no engine reads a half-written file. engines are the replicas’ root URLs, not their /v1 API URLs, and timeout bounds each HTTP call in seconds. The push fails if a call answers with any status but 200, or, for a call that reports its outcome, with a success that is not true. Both engines report some failures as a 200 with {"success": false}.

For vLLM (engine="vllm", v0.30.0, VLLM_SERVER_DEV_MODE=1), the push reloads the directory through /collective_rpc inside the publication lifecycle it shares with NCCLPush: every replica pauses with mode=wait, takes the weights, resets its prefix cache, sets the weight version and resumes, and a phase that fails on any replica stops the push before the next phase, leaving the paused replicas paused until a later push succeeds. In-flight draws therefore finish entirely on the old weights, and no cached prefix outlives them. For SGLang (engine="sglang", v0.5.20), the push is one /update_weights_from_disk call, which waits for in-flight requests, holds new ones and flushes the radix cache. A failed SGLang load rolls back by re-reading the same directory.

A failed push raises an error that names every replica that did not take the version. Every process of a pool calls the push: each runs the same writer over the served tree and every other tree the source writes beside it (a decision release’s readout, _published), reading each tensor home together as it is written (reading_home), and process 0 alone writes the weights and publishes them; an agreement point raises a failure on every process, so the others do not hang at the next collective.

def write(variables: Variables) -> None

Write variables into directory, replacing each file atomically.

Every process of a pool calls it.

class source

class Scoring(Protocol)

A task served by forward passes over its requests’ rows (ScoringServer). It holds no weights of its own: the server holds them resident, and hands them to every pass, so a task and the weights it scores with are one pair the server swaps whole (ScoringServer.reload).

def admit(request: Request) -> ScoringRequest[Row, Payload]

request laid out as the rows the task’s passes read; raises ValueError for a request it refuses.

def planned() -> PassPlan[Row]

An empty pass of the task’s, which takes rows one at a time while they are one pass: within its budget, and alike enough for one compiled forward.

def forward(variables: Variables, rows: Sequence[Row]) -> Sequence[Output]

One pass over rows with variables: each row’s output, in order.

def compile(variables: Variables) -> Sequence[object]

Compile every program the task’s passes run with variables, up front, so a server compiles none while it serves; returns the shapes it serves at, one program each.

def answer(payload: Payload, outputs: Sequence[Output]) -> Answer

A request’s answer from what it admitted and its rows’ outputs, in its rows’ order.

dataclass source

class ScoringRequest(rows: tuple[Row, ...], payload: Payload)

A request as its task admitted it: its rows, each scored in one of the task’s passes, and what the task’s answer reads of it besides their outputs.

class source

class ScoringServer(
task: Scoring[Request, Row, Payload, Output, Answer],
variables: Variables,
*,
rows: int | None = None,
)

Answers scoring requests through one task’s passes, one pass a step.

A pass takes the waiting rows oldest first, as many as the task’s plan takes (planned), all laid out by one task: a request admitted before a reload is scored by the task and weights that admitted it. A step looks at every waiting request while the pass’s plan takes their rows, or at most rows, cancelled ones among them: a pass that looks no further than a few requests leaves the windows of a larger shape part empty: over a set of 17.5M real tokens, passes of 64 requests at most computed 22.3M tokens in 389 passes, passes over every waiting request 20.1M in 308, the same programs. Build one with from_task.

arrived: Callable[[], None]

Called after each submission: an Engine serving this kind wakes to it.

pooled

Whether its weights’ mesh spans several processes (Kind.pooled), which an Engine, stepping a kind of one process, refuses.

programs: tuple[object, ...]

The shapes of the programs compiled up front (from_task), one each.

busy: bool

Whether a request waits for a pass, or for the rest of its rows.

def from_task(
task: Scoring[Request, Row, Payload, Output, Answer],
variables: Variables,
*,
rows: int | None = None,
) -> ScoringServer[Request, Row, Payload, Output, Answer]

A server over task’s passes with variables, held resident, every program its passes run compiled up front (Scoring.compile, the shapes in programs); a step looks at every waiting request, or at most rows. A server built directly compiles each program on its first pass.

def metrics() -> dict[str, float]

What the task it serves reports of its passes so far (Metered), the share of their tokens its rows held among them; none for a task that reports nothing.

def submit(request: Request) -> Timed[Answer]

Queue one request and return its future answer. The task lays it out here, so a request it refuses raises its ValueError here, and the others go on.

def step() -> None

Run one pass over the waiting rows, and answer every request whose last row it scores.

def run() -> None

Step until no request waits.

def reload(
task: Scoring[Request, Row, Payload, Output, Answer],
variables: Variables,
) -> None

Serve task with variables in place of the current pair, both at once, for every request submitted after: a request is scored by the pair that laid it out, so its answer pairs one task with its own weights, as a task’s calibration is fitted to the weights it came with. The tree must match the served one (dew.inference.engine.normalized), and its leaves are copied onto the served placement before the swap. Safe against a step on another thread.

def fail(
failure: BaseException,
taken: Sequence[_Request[Row, Payload, Output, Answer]] = (),
) -> None

Stop serving after failure: every waiting request, and taken’s, settles with it, and a later submission or step is refused.

class source

class Server(
model: nn.Module,
variables: Variables,
processor: Processor | None,
*,
sampling: Sampling,
transforms: tuple[LogitsTransform, ...],
stopping: Finishing,
grammar: Grammar | None,
rows: Rows,
slots: int,
capacity: int,
admission: int,
default_budget: int | None,
max_length: int | None,
decode_steps: int,
prefix: int | Literal['prompt'] | None = None,
precision: Numerics | None = None,
progress: Progress = END,
prefill_mesh: Mesh | None = None,
)

Serves text requests with continuous batching over one resident KV cache.

A TextGeneration call runs its batch as one program, so no request can join until it returns. A server keeps slots rows of capacity cache slots each and refills them from a queue. Each step prefills the prompts admitted that step into free rows and draws one token for every occupied row, or a speculative block where the task speculates (from_task). A row leaves where the task’s criteria end it (EOS, a stop string) or at its budget, and the next request takes its slot. Every request runs the task’s policy and draws the same tokens it would draw through the task alone with the same key, except where the model’s products round at boundaries that the two programs’ roundings cross (the module docstring).

Build one with from_task. admission bounds the prompts one step prefills. A request whose prompt and budget do not fit the capacity is refused at submit, with the error TextGeneration raises for a request over the model’s context. Over a mesh, the slots and the admission split evenly over the groups of rows, and so do the pages of a paged pool. The server holds the weights at their decoding storage (dew.interop.weights.decoding_storage), to which reload casts the weights it is handed.

cross: tuple[CrossBlocks, int, np.dtype] | None

The model’s cross-attention layers and its context’s width and dtype, where a request may carry a context beside its prompt.

arrived: Callable[[], None]

Called after each submission: an Engine serving this kind wakes to it (dew.inference.engine.Kind).

prefix_hits

The number of prompt tokens served from shared prefix pages instead of prefilled.

pooled

Whether the server’s mesh spans several processes, which step it together (run, step); an Engine steps a server of one process.

group_fields

The group fields some layer reads whole (dew.nn.kv_cache.GROUPS), whose groups its cache keeps beside the keys: the ones a served prompt’s slot beside its tokens.

keyed

Whether a layer’s decode cache holds its rows’ rounding keys (dew.nn.kv_cache.ROUNDING_KEYS), so a prompt’s state follows its request’s key, which then names its prefix too (_prepared).

mixed_refusal

Why the admitting step prefills in a separate forward, or None when it runs one mixed forward.

The mixed forward covers each row’s last draw and the admitted prompts in one pass, so the projections read their weights once.

cache: Variables

The resident cache, updated in place by every step.

occupancy: int

The number of rows the host knows to be running, one step behind the device.

busy: bool

Whether a request waits or runs, or a call’s draws wait to be read.

def from_task(
task: TextGeneration,
*,
slots: int,
capacity: int,
admission: int | None = None,
kv_cache: KVCache | None = None,
chunk: int | None = None,
prefix_cache: bool = False,
prefix_states: int | None = None,
prefix_state_every: int | None = None,
host_pages: int = 0,
decode_steps: int = 1,
prefill_mesh: Mesh | None = None,
) -> Server

Build a server over the task’s model, weights, processor and policy.

capacity is rounded up to whole 64-slot tiles and whole pages, and must stay within the model’s context. The task’s n must be one, and its strategy the row-wise sampler, with or without a grammar, or speculative blocks (Speculative): each iteration then drafts, verifies and emits a block in every row, which draws as the row-wise sampler draws, over a dense cache whose prompts prefill whole.

kv_cache replaces the model’s cache layout (dew.nn.kv_cache). A paged layout pools the pages of every row, so short requests fit more rows than pages * page_size / capacity. chunk prefills a prompt in pieces of at most that many tokens, one piece a step, so a long prompt does not stall the rows beside it. It also works over a dense cache where the admitting step is mixed. prefix_cache lets a request reuse the pages of a prompt prefix an earlier request computed, and needs a paged cache. Where the model keeps a row’s own state beside its pages (a recurrent mixer’s memory, latent attention’s index keys), a reused prefix is read with the state its prompt left at its end: each group keeps prefix_states of them (one a slot by default), taken at a prompt’s deepest shareable page boundary and every prefix_state_every pages below it, a piece cut to end there, and a request reuses pages as deep as a state is kept (dew.inference.pages.States). host_pages keeps that many more prefix pages in host memory, split over the groups: a cached page the pool reclaims goes to the host first, and a prompt that shares it reads it back into a fresh page before its prefill (dew.inference.pages), so a prefix cache holds more than the device’s pages. Where rows keep a state beside their pages, a state giving way goes to the host too, as many states as the bytes of host_pages pages hold.

prefill_mesh prefills the admitted prompts on a mesh of their own, as MaxEngine’s and vLLM’s disaggregated serving does: the server keeps a copy of its weights there, placed as on its own mesh, prefills each admission’s rows over it and moves their decoder state to the decoding mesh, where the rows are seated and decode; a request draws what it draws on one mesh. The rows prefill whole over a dense cache.

Weights on a mesh are served on that mesh. The slots, the admission and the pages are split over its row axes, so the number of row groups must divide the slots and the admission. The admission defaults to the largest multiple of the group count up to eight rows an iteration. The model decodes its stack and sequences whole on a mesh with a stage or sequence axis (dew.nn.sharding.whole). On a mesh over several processes each process seats and reads its own groups’ rows and the pool steps the server together (run).

decode_steps runs that many iterations per device call, so the host launches the step’s kernels once per call; on a tensor axis those launches take about as long as the step. Requests are seated, and draws read, only at call boundaries, so a request waits up to decode_steps iterations for its slot. The default admission seats that many iterations’ rows per call. The draws are the same for any value. More iterations pay off where the host’s launches leave the devices idle, as across an NVLink pair at 128 slots (docs/concepts/inference.md). On one RTX 4080, which was already busy, more iterations were slower at 32 and 128 slots, and at 64 slots only two iterations gained, by under 1%. A call seats requests only before its first iteration, so the slots take more iterations to fill (docs/performance.md).

A server draws at the progress its task draws at (TextGeneration.progress), the run’s end or a run’s, and reload moves it to the version it serves.

def reload(variables: Variables, progress: Progress | None = None) -> None

Serve variables from the next step on, in place of the current weights, at the run’s progress where one is given (a run’s version, NativeRolloutServer.load), else at the one it served. A progress takes the place of one of its kind: a server built at the run’s end serves that end, and one built at a run’s progress serves a run’s progress, its steps compiled for it.

The tree must match the served one leaf for leaf in shape, and a leaf that is not floating point must match its dtype too. A floating leaf is cast to the served precision, so the compiled step keeps running without a recompile. The leaves are copied onto the served placement, so the caller may donate its buffers right after. Running rows keep their cache and draw their next token from the new weights, and prefix pages the old weights wrote are no longer shared. reload is not thread-safe against step, so the caller must serialize the two. A server drawing speculative blocks reloads between requests alone: a running row’s next block opens on the boundary logits its last block verified under the old weights, which no next-step draw recomputes.

def submit(
prompt: Prompt,
max_new_tokens: int | None = None,
*,
key: int | jax.Array,
adapter: str | None = None,
shared: int | None = None,
) -> Ticket

Queue one request and return a ticket that resolves to its Generation.

prompt is one text prompt, ModelInputs row or token row, without media or noncanonical positions; a ModelInputs row may carry the context a cross-attention decoder reads (dew.nn.inputs.CONTEXT) and the image and block groups its layers read whole (dew.nn.kv_cache.GROUPS), which slot beside its tokens. It is validated as TextGeneration validates it, against the server’s capacity in place of the model’s context. Without max_new_tokens, the task’s default budget applies, its max_new_tokens or what its max_length leaves past the prompt. A request with a zero budget resolves at once, with the prompt alone. Another thread may submit while one steps. The request is validated here and queued at once; a step seats it when a slot is free. adapter names the request’s adapter where the task’s model is an adapter set’s (dew.lora.AdapterSet), each row drawing under its own: a request draws what a task over that adapter alone draws, and the base model’s where it names none. shared declares the prompt’s first shared tokens a segment other requests share, wherever it ends (a decision’s encoded state, each question past it a request of its own): a paged server with prefix_cache keeps the keys and the row state at its end, and a request whose prompt starts with the same segment, declaring it too, prefills only its rest, the page the segment ends inside copied into its own (dew.inference.pages.segment_hash). Such requests submitted together wait for the first to write it. A segment ends between two tokens no group (a block, an image) holds both of.

def step() -> None

Run one device call and read the previous call’s draws.

The call admits the queued requests that fit and runs decode_steps iterations. Its own draws are read by the next step, or by run.

def run() -> None

Step until every queued and running request has resolved. A step that fails stops the server: every request it holds settles with the failure (fail), which run raises.

def fail(failure: BaseException) -> None

Stop after failure: every waiting and running request settles with it, and a later submission or step is refused.

dataclass source

class TextToImage(
model: nn.Module,
process: Process,
inputs: InputSpec,
variables: Variables,
autoencoder: AutoEncoder | None = None,
steps: int = 50,
guidance: Guidance | None = None,
solver: Solver[object] = DDIM(),
grid: Callable[[int], tuple[Process, jax.Array]] | None = None,
final_denoise: bool = True,
finish: Callable[[Variables, jax.Array], jax.Array] | None = None,
*,
precision: Numerics | None = None,
)

Generates images from text prompts with a trained diffusion model and its encoders.

Call it as pipe(prompts, key=0) or pipe(prompts, steps=40, guidance=4.0, solver=Heun(), key=key).

variables is the objective’s whole tree, with the EMA copy merged over the live weights when the run kept one, so a sample comes from the weights the run publishes. steps, guidance and solver are the defaults for a call that omits them; an objective or a loaded source sets them. grid is for a source whose solver pairs its own sigma and model-time tables, and returns the process and its explicit time grid for a step count. With final_denoise False a trajectory ends the way those solvers end it. finish runs on the decoded images under the same placement, for a source that ships a checker or an output transform.

The weights stay where they are placed. On a mesh, the prompts are split into per-process rows over its batch axes, and the result keeps that sharding. Each row’s initial noise comes from its global row index, so a pool of processes draws the same noise as one process does for the same prompts.

latent_shape: tuple[int, ...]

The per-example shape the model denoises.

It is the sample field’s shape, or that field’s latent shape when the task has an autoencoder in front of the model.

def bind(variables: Variables) -> TextToImage

Return this task with variables as its weights, its encoders’ included: each call encodes its prompts and its blank with the towers it holds.

def quantized(spec: Quantization) -> TextToImage

Return this task with its denoiser’s weights quantized as spec says.

The denoiser’s matmuls then compute with the quantized weights (dew.quantization.quantize_for_serving). The encoders and the autoencoder keep their weights.

def from_objective(objective: DiffusionObjective, variables: Variables) -> TextToImage

Build a task over the objective’s model and variables that samples the way its evaluation does.

A loss-only head the objective trains is dropped.

def from_saved(
run: SavedRun,
*,
ema: bool | None = None,
mesh: MeshSpec | Mesh | None = None,
layout: Layout | None = None,
dtype: DTypeLike | None = None,
param_dtype: DTypeLike | None = None,
variables: Variables | None = None,
) -> TextToImage

Load the run run (its checkpoint pinned, SavedRun.pinned), built from the inference record its checkpoint holds (Checkpoints.artifact) the way the recipe built it; the run’s run.json is not read. from_run loads one from its directory.

ema None reads the averaged weights when the run kept them and the live ones when it kept none; True requires the averaged weights, and False reads the live ones. A run whose averaged weights are a reference policy, such as Flow-GRPO’s frozen KL reference, always reads its live policy. With mesh, the weights are restored directly onto that mesh under layout, the way the trainer places them. Without it, they go onto a default MeshSpec() over the current pool’s devices.

dtype sets the compute dtype of the model, the encoders and the VAE. param_dtype sets the dtype the parameters are stored in; None keeps the checkpoint’s dtypes exactly. Supplied variables, a tree of the run’s own form (a run that started from it saved them), are bound in place of the checkpoint’s, which is not read.

def from_flaxdiff(
directory: str | os.PathLike,
config: Mapping[str, object],
*,
jax_version: str,
ema: bool = True,
best: bool = False,
dtype: DTypeLike | None = None,
) -> TextToImage

Load a FlaxDiff text-to-image run (simple_udit or hybrid_dit on the SD VAE) into Dew’s model.

directory is one checkpoint step, config the run config FlaxDiff’s trainer logged, and jax_version the jax version the run trained under, from its requirements.txt. ema and best pick the weights; dtype is the model’s compute dtype. dew.interop.flaxdiff reads the format.

def prepared_process(steps: int) -> tuple[Process, tuple[float, ...] | None]

Return the process and explicit time grid that a call with steps steps runs.

The grid is a tuple of concrete times, so the compiled trajectory has a fixed length and fixed values. It is None when the task has no grid. Raises ValueError unless steps is a positive int.

def prepare(
prompts: str | Sequence[str | Mapping[str, object]] | None = None,
*,
conditions: Mapping[str, Conditioning] | None = None,
key: int | jax.Array,
steps: int | None = None,
unconditional: str | Sequence[str | Mapping[str, object]] | Mapping[str, Conditioning] | None = None,
image: ArrayLike | None = None,
image_latents: ArrayLike | None = None,
mask: ArrayLike | None = None,
noise: ArrayLike | None = None,
initial: ArrayLike | None = None,
times: ArrayLike | Sequence[float] | None = None,
encode_key: int | jax.Array | None = None,
) -> DenoisingInputs

Encode the conditions and build the initial state for a call, on a concrete grid.

Pass either prompts or conditions. The result is DenoisingInputs, which a call takes in place of prompts. Each array argument has one row or one row per sample. The optional inputs:

  • image: uint8 or normalized floating NHWC pixels at the task’s geometry.
  • image_latents: the image already encoded, which skips VAE encoding.
  • mask: spatial conditioning added to both guidance branches; it needs image and an autoencoder.
  • noise: unit Gaussian noise for noising a clean image.
  • initial: an already-noisy latent state for a continuation or refiner handoff, which is never noised again.
  • times: an explicit grid that selects a partial trajectory in the prepared process.
  • encode_key: the key that samples a VAE posterior; None uses its mean.

conditions are prompts already encoded, {keyword: condition} with one row per sample. A pipeline’s conditioner loaded on its own can encode them, so the text encoder does not have to share the device with the denoiser (load_diffusion_source(text=False)). unconditional encoded the same way, one row or one per sample, is the branch guidance reads. It can also be negative prompts as text, which the text encoder encodes. Without it, the task’s own blank prompt is encoded if the text encoder is loaded. If it is not loaded, the result has no unconditional branch, and a call with it must pass guidance=None. Each row’s noise is the noise a call with prompts draws.

Raises ValueError for a combination the task cannot run, such as image together with image_latents, or noise without a clean image.

def scoring(
*,
steps: int | None = None,
guidance: Guidance | float | None | Omitted = OMITTED,
solver: Solver | None = None,
decode: bool = True,
images: int = 8,
) -> tuple[ImageScoring, Variables]

This task as a scoring task of ImageRequests (dew.inference.scoring.Scoring), which a ScoringServer serves beside the other kinds of an Engine, and the weights it samples with. steps, guidance and solver are a request’s where it names none, the task’s where these are omitted. A request is answered with the Images its lone call task(prompts, key=key, ...) draws with its settings: its noise and its walk’s draws are its key’s, whatever requests share its pass. A pass holds requests alike (as many prompts and negatives, the same settings) up to images images, and walks them as one program mapped over the requests, each request’s walk its own. The scoring task holds no weights: a server holds them resident and swaps both whole on a reload (ScoringServer.reload).

class source

class Timed()

A request’s future, with when it was submitted, admitted to a step and finished, in time.perf_counter seconds; admitted and finished are None until they happen.

dataclass source

class Usage(
prompt_tokens: int | None = None,
completion_tokens: int | None = None,
total_tokens: int | None = None,
)

Token usage the backend reported for the whole request.

Each count is None when the backend did not report it.

class source

class VLLMGenerateServer(
base_url: str,
sampling: Sampling,
weights: WeightSync,
*,
version: int = 0,
workers: int = 64,
routing: bool = False,
timeout: float = 600.0,
)

Serves rollouts from vLLM’s token route, POST /inference/v1/generate.

This is the one vLLM route that returns, beside the sampled ids and their likelihoods, the ids the sampler kept for each of them (GenerateResponseChoice.sampling_mask). So a top-k or top-p policy trains on its recorded support (support_log_probs). Start the engine with:

  • --enable-scale-out (or --tokens-only);
  • --return-sampling-mask, which needs Model Runner V2 and no speculative decoding;
  • --logprobs-mode processed_logprobs, so the reported likelihoods are the filtered ones;
  • --enable-return-routed-experts when routing is set.

vLLM builds the mask only under a finite top-k, so a Sampling without top_k raises ValueError. The route draws under temperature, top-k, top-p and min-p alone, so a Sampling that sets a repetition, presence or frequency penalty, no_repeat_ngram_size, min_new_tokens, typical_p or stop raises ValueError too.

function source

def pipeline(
source: str,
*,
mesh: MeshSpec | Mesh | None = None,
layout: Layout | None = None,
dtype: DTypeLike | None = None,
param_dtype: DTypeLike | Literal['auto'] | None = None,
ema: bool | None = None,
step: int | str | None = None,
revision: str | None = None,
trust: Sequence[str] = (),
options: SourceOptions | None = None,
precision: Numerics | str | None = None,
) -> TextToImage | TextGeneration | BlockGeneration | MaskedGeneration | SavedTask

Load the inference task for source, with its weights placed once.

source is a run directory, a source checkpoint directory or a Hub repository holding either. A run loads as the task its recorded objective declares (Objective.saved_task), including a plugin objective’s own task. mesh places the weights on that mesh under layout, or under the trainer’s default layout when layout is None. Without mesh, data parallelism uses the current pool’s devices.

dtype sets the computation dtype, given as a dtype (jnp.bfloat16) or its name. param_dtype sets parameter storage. None and 'auto' keep each tensor’s stored dtype, a run’s or a source’s (a quantized source’s packed weights decode to the dtype it declares); jnp.float32 gives a source FP32 masters. ema selects a run’s averaged weights: None reads them when the run kept them and its live weights otherwise, and True always reads them. step selects a run’s checkpoint and revision pins a Hub source; passing revision for a run directory or step for a source raises ValueError. trust names the packages outside Dew a run’s record may import, and the sources whose own code a load may run (a media processor’s, LFM3-VL’s), as trust_remote_code does in transformers. options selects what a source builds, as a run’s --source options do (SourceOptions: a wrapper’s language model alone, the prediction depths kept, each layer kind’s attention); a run’s record selects its own, so passing them for a run directory raises ValueError. precision gives a source’s task its numerics (Pretrained.load(precision=...): its dtypes, and the precision and forms its calls trace under); a run’s record names its own, so passing it for a run directory raises ValueError.

Loading a task also points XLA at the on-disk executable cache, unless a cache directory is already set, so a restarted process reuses what it already compiled.