dew.inference
Inference tasks: native generation with a model and its weights, and clients for external engines, each imported only when you use its API.
So a text-to-image task does not import the run loader’s checkpointing, the data pipeline or the rollout servers.
| Name | Summary |
|---|---|
BlockGeneration | Generates block-diffusion canvases from a block denoiser and its weights. Documented in dew.inference.tasks. |
CheckpointBanks | Serves banks by restoring a run’s published weights one bank at a time. Documented in dew.inference.banks. |
Completion | The choices of one completion request in prompt-major order, with the SDK responses kept. |
DenoisingInputs | Encoded conditioning and initial noise, placed on the devices the way a call uses them. |
Draw | One model call as the engine served it: the one record a server, an environment, a session and the packer read. |
Engine | Serves kinds from one thread of its own between start and close: each turn, every busy kind takes one step, so a kind’s long queue never starves another’s; with none busy the thread waits for a submission. |
HeldBanks | Serves banks from a variables tree already held in memory. Documented in dew.inference.banks. |
ImageRequest | One request a served text-to-image task answers (TextToImage.scoring): its prompts, an image each, the key its draws come from, and the settings of its lone call task(prompts, key=key, ...): steps, guidance and solver (the scoring task’s where omitted), and negative, its negative prompts, one or one a prompt, in place of the task’s blank. |
Images | Decoded samples in [-1, 1], NHWC (NTHWC for a video field), placed as the task ran them. |
InfrastructureError | A rollout’s infrastructure failed: a connection, an engine, a sandbox, a runtime or a service. |
Kind | One serving kind, as Engine drives it. |
LayerBanks | Serves a banked store’s values, read one bank at a time. Documented in dew.inference.banks. |
MaskedGeneration | Samples a response with native MDLM, holding the prompt fixed: the whole response at once (Unmask), or a block of block places at a time as LLaDA2’s releases walk (DiscreteProcess.generate’s block, Confident’s reveal). Documented in dew.inference.tasks. |
NCCLPush | Publishes a policy version to vLLM replicas by NCCL broadcast from a trainer GPU. |
NativeRolloutServer | Serves rollouts from Dew’s Server, stepped by an Engine of its own, or where the server’s mesh spans several processes (pooled) by the run itself, every process together (waited). |
OllamaCompletion | Calls one Ollama model through an ollama.Client or ollama.AsyncClient that you pass in. |
OpenAICompletion | Calls one model through an OpenAI client, including one pointed at a vLLM or SGLang base_url. |
OpenAIRolloutServer | Serves rollouts from the OpenAI-compatible completions endpoint of a vLLM or SGLang engine. |
Processor | Interface for a loaded source’s host processor, which prepares requests and decodes tokens. Documented in dew.inference.tasks. |
Publication | Publishes weights to an engine fleet under a version, for use as a scheduler’s Publisher. |
RolloutServer | Samples continuations of token prompts under a versioned policy whose weights can be reloaded. |
RunProcessor | Adapts a run’s tokenizer to the Processor interface a task uses. |
SafetensorsBanks | Reads a local HF decoder checkpoint without materializing its layer stack. Documented in dew.inference.banks. |
SafetensorsReload | Publishes a policy version to engine replicas through safetensors on disk. |
Scoring | A task served by forward passes over its requests’ rows (ScoringServer). |
ScoringRequest | A request as its task admitted it: its rows, each scored in one of the task’s passes, and what the task’s answer reads of it besides their outputs. |
ScoringServer | Answers scoring requests through one task’s passes, one pass a step. |
Server | Serves text requests with continuous batching over one resident KV cache. |
TextGeneration | Generates next tokens from a decoder, its weights and its processor. Documented in dew.inference.tasks. |
TextToImage | Generates images from text prompts with a trained diffusion model and its encoders. |
Timed | A request’s future, with when it was submitted, admitted to a step and finished, in time.perf_counter seconds; admitted and finished are None until they happen. |
Usage | Token usage the backend reported for the whole request. |
VLLMGenerateServer | Serves rollouts from vLLM’s token route, POST /inference/v1/generate. |
pipeline | Load the inference task for source, with its weights placed once. |
Completion
Section titled “Completion”class Completion( texts: tuple[str, ...], finish_reasons: tuple[str | None, ...], token_counts: tuple[int | None, ...], usage: Usage | None, responses: tuple[OllamaResponse | OpenAIResponse, ...], tokens: tuple[tuple[int, ...] | None, ...], log_probs: tuple[tuple[float, ...] | None, ...], routed_experts: tuple[np.ndarray | None, ...] = (),)The choices of one completion request in prompt-major order, with the SDK responses kept.
token_counts holds per-choice counts, which stay None when only
aggregate usage was reported. finish_reasons keeps the backend’s
values, including an absent reason. tokens and log_probs hold each
choice’s sampled ids and their log-probabilities where the backend
reported them; a vLLM completion asked for logprobs with
return_tokens_as_token_ids in extra_body reports both. The
log-probabilities come from whatever distribution the engine was
configured to report, and this record does not relabel them.
routed_experts holds each choice’s [forwarded ids, layers, top_k]
expert record when vLLM ran with --enable-return-routed-experts.
DenoisingInputs
Section titled “DenoisingInputs”class DenoisingInputs( conditions: Mapping[str, Conditioning] = struct.field(default_factory=dict), unconditional: Mapping[str, Conditioning] = struct.field(default_factory=dict), rows: int | None = struct.field(pytree_node=False, default=None), grid_steps: int | None = struct.field(pytree_node=False, default=None), process: Process | None = struct.field(pytree_node=False, default=None), times: tuple[float, ...] | None = struct.field(pytree_node=False, default=None),)Encoded conditioning and initial noise, placed on the devices the way a call uses them.
TextToImage.prepare builds them, and a TextToImage call takes them in
place of prompts. rows is the number of this process’s real prompts. On
a mesh the arrays hold the padded batch, sharded by row, which a call uses
as it is. A call refuses inputs prepared for another step count, another
geometry or another mesh.
class Draw( prompt: tuple[int, ...], tokens: tuple[int, ...], behavior_log_probs: tuple[float, ...], finish_reason: str, version: int, raw_log_probs: tuple[float, ...] | None = None, routed_experts: Routing | None = None, support: tuple[tuple[int, ...], ...] | None = None,)One model call as the engine served it: the one record a server, an environment, a session and the packer read.
prompt are the ids the engine read and tokens the ids it sampled,
with the id that ended the draw included when it stopped: an EOS id, or
the id completing a stop string (check_ending). behavior_log_probs
is each sampled id’s likelihood under the distribution that drew it, and
raw_log_probs the unmodified model’s, or None where the backend cannot
report it (vLLM, SGLang, a recording gateway). finish_reason is one of
FINISH_REASONS, and the draw terminated where it is stop or
tool_calls. version is the policy version being served when the
request was submitted, the oldest policy that may have produced any of
its ids.
Two engine records are optional. routed_experts is the mixture’s
routing for every id the engine forwarded, as
[len(prompt) + len(tokens) - 1, layers, top_k] expert ids in the dtype
the engine sent; the last sampled id is never forwarded. vLLM calls this
record routed_experts and SGLang meta_info.routed_experts, and the
trainer replays it (dew.nn.protocols.Routes). A model whose layers
route at several sites, a routed mixer beside a routed feed-forward,
records one such array a site, keyed by its path (Routing). support holds, for
each sampled id, the token ids the sampler’s top-k/top-p filters kept
(vLLM’s sampling_mask). With a support, the behavior log-probability
is the filtered one, and the trainer renormalizes over the same support.
None means no filter. Ids, likelihoods and records that do not match the
sampled ids one to one, or are not finite, raise ValueError.
terminated: bool-
Whether the draw ended itself, on an EOS id, a stop string or a tool call, rather than being cut at its budget or aborted.
Draw.checked
Section titled “Draw.checked”def checked(sampling: Sampling) -> DrawAdmit a nonempty sampled action ending as sampling ends one (check_ending).
Imported call records may be empty, but an action presented by a sampling server may not.
Draw.response
Section titled “Draw.response”def response(sampling: Sampling) -> tuple[int, ...]The drawn ids a reward reads decoded (response).
Engine
Section titled “Engine”class Engine(*kinds: Kind = (), name: str = 'serving-engine')Serves kinds from one thread of its own between start and close: each turn, every busy
kind takes one step, so a kind’s long queue never starves another’s; with none busy the thread
waits for a submission. A step that fails stops every kind, each request settled with the
failure, as one device’s failure is every kind’s.
A request’s callback runs on the serving thread, inside its kind’s step, and may reload or
close from there. A program that ends without close stops the thread after the step in
progress (dew._coordination.stop_at_exit).
Engine.start
Section titled “Engine.start”def start() -> NoneEngine.close
Section titled “Engine.close”def close() -> NoneStop serving once what waits is answered.
Engine.stop
Section titled “Engine.stop”def stop() -> NoneStop serving after the step in progress; what waits is left to its kind’s owner.
Engine.reload
Section titled “Engine.reload”def reload(reloading: Callable[[], None]) -> NoneRun reloading, a kind’s reload, between two steps.
ImageRequest
Section titled “ImageRequest”class ImageRequest( prompts: str | Sequence[str], key: int | jax.Array, steps: int | None = None, guidance: Guidance | float | None | Omitted = OMITTED, solver: Solver | None = None, negative: str | Sequence[str] | None = None, prepare_key: int | jax.Array | None = None, prepare_steps: int | None = None, on_step: Callable[[int, np.ndarray], None] | None = None,)One request a served text-to-image task answers (TextToImage.scoring): its prompts, an image
each, the key its draws come from, and the settings of its lone call task(prompts, key=key, ...):
steps, guidance and solver (the scoring task’s where omitted), and negative, its negative
prompts, one or one a prompt, in place of the task’s blank. A request whose noise and conditions are
prepared apart from its walk, as task(task.prepare(prompts, key=prepare_key, steps=prepare_steps, unconditional=negative), key=key, steps=steps, ...) prepares them, names prepare_key and
prepare_steps (its key and steps where None). on_step, where given, streams the walk: it is
called on the runtime’s callback thread with each solver step’s number, from 1, and the clean
prediction that step reads, the request’s rows as a host array, as the walk runs.
Images
Section titled “Images”class Images( rows: int | None = struct.field(pytree_node=False, default=None), latents: ArrayT | None = None,)Decoded samples in [-1, 1], NHWC (NTHWC for a video field), placed as the task ran them.
latents holds the denoised latents, and images is None when the
call ran with decode=False. host() copies this process’s rows
real rows back to host arrays, and pil() returns an image batch’s real
rows as 8-bit images.
Images.host
Section titled “Images.host”def host() -> Images[np.ndarray]Return this process’s real rows as host arrays, without the padding added to fill the devices.
Images.pil
Section titled “Images.pil”def pil() -> list[PILImage]Return this process’s real rows of an NHWC image batch as PIL images.
Three channels give RGB images and one channel gives grayscale. The
pixels are quantized by dew.artifacts.uint8_pixels. Raises ValueError
when the samples kept only their latents, or when they are not
[N, H, W, 3] or [N, H, W, 1] images.
InfrastructureError
Section titled “InfrastructureError”class InfrastructureError(RuntimeError)A rollout’s infrastructure failed: a connection, an engine, a sandbox, a runtime or a service.
A session source turns it, like any OSError, into an INFRA_ERROR
session, which the scheduler retries; any other error is broken code or
a broken contract and fails the rollout. The remote rollout servers
raise it when the engine cannot be reached, times out, or answers 5xx
or 429, and Dew’s sandbox runners when they cannot run a program. An
environment, verifier or reward raises it for a failure of its own
external boundary that is not an OSError, such as an httpx transport
error.
class Kind(Protocol)One serving kind, as Engine drives it.
arrived: Callable[[], None]-
Called after each submission, so an engine waiting for work wakes to it.
pooled: bool-
Whether its steps span several processes, which step it together (
dew.inference.Server.run); an engine steps a kind of one process, as only that process’s submissions wake it. busy: bool-
Whether a request waits for, or runs in, a step.
Kind.step
Section titled “Kind.step”def step() -> NoneOne bounded device call: admit what fits, run it, settle what finished.
Kind.fail
Section titled “Kind.fail”def fail(failure: BaseException) -> NoneStop after failure: every waiting and running request settles with it.
NCCLPush
Section titled “NCCLPush”class NCCLPush( source: Pretrained, engines: tuple[str, ...], library: str | Callable[[int], Collectives], dtype: str = 'bfloat16', chunk: int = 512 * 2 ** 20, timeout: float = 600.0,)Publishes a policy version to vLLM replicas by NCCL broadcast from a trainer GPU.
The replicas run vLLM (checked against v0.30.0) launched with
--weight-transfer-config '{"backend": "nccl"}' and
VLLM_SERVER_DEV_MODE=1. The module docstring of dew.inference.nccl
describes the handshake and the push in full.
sourceis thePretrainedthe replicas were launched from; itsexportgives the tensors, of the trained tree and of every other tree the source exports (a decision release’s readout), each read home by the pool as process 0 sends it (reading_home).enginesare the replicas’ root URLs, not their/v1API URLs.libraryis the engine’s own libnccl.so.2, because NCCL refuses a peer of another version when the group opens; or a maker of the push’sCollectivesfrom the sending device’s CUDA ordinal, which makes its calls instead (an emulation of NCCL over the host).chunkbounds the bytes of tensors placed on the sender device beside the trainer’s state. The first chunk is read before any replica pauses, and each next one is read and copied over on a reader thread while the current one broadcasts, so a push holds at most two chunks there, or two of its largest tensor.
Floating-point leaves are cast to dtype before the push, and timeout
bounds each HTTP call in seconds. Groups open on the first push and stay
open until close. Every process of a pool calls the push.
timings holds the seconds each phase of the last push took: export
(the publication’s specs and chunks, before any tensor is read, on every
process), and on process 0, which sends, send (reads, copies and
broadcasts until every replica has loaded the tensors) and total.
NCCLPush.close
Section titled “NCCLPush.close”def close() -> NoneAbort the NCCL groups this side opened. A later push opens them again.
Groups are aborted rather than destroyed, because NCCL’s destroy
finalizes the group, which waits on the engine’s side, and an engine
keeps its side open until it exits. A failed push also calls close,
because a group whose broadcast or replica failed cannot deliver the
next version.
NativeRolloutServer
Section titled “NativeRolloutServer”class NativeRolloutServer(server: Server, *, version: int = 0)Serves rollouts from Dew’s Server, stepped by an Engine of its own, or where the server’s mesh
spans several processes (pooled) by the run itself, every process together (waited).
The engine orders weight loads with the server’s steps, so a load happens
between two steps. A submission waits for a load in progress but not for a
step. The server takes it while a step runs and seats it at the next one.
The server’s own sampling policy and transforms decide the draws, and a
draw keeps both the raw and the behavior likelihoods the server records.
The engine’s thread starts at the first submission, so constructing the
server starts nothing. If the program ends without close, the engine
stops after the step in progress.
A draw ends on one of the policy’s EOS ids or on the id completing one of
its stop strings, and Draw.terminated tells those from a draw cut at its
budget or by a criterion that truncates it (MaxNewTokens). A server
whose task ends rows on a criterion of its own besides those, which a
draw’s policy cannot name, raises ValueError.
deferred builds one whose server is built from the first version it is
loaded with, as a run that builds its rollout before its weights are drawn
serves them (RolloutScheduler.served).
engine: Engine | None-
The engine that steps the server, and orders a load with its steps; None for a deferred server not yet loaded, and for a pooled one, which the run steps (
waited). pooled: bool-
Whether the server’s mesh spans several processes, which step it together (
waited).
NativeRolloutServer.deferred
Section titled “NativeRolloutServer.deferred”def deferred( build: Callable[[Variables, int], Server], sampling: Sampling,) -> NativeRolloutServerA server that draws nothing until its first load, which builds its Server with build from
the loaded variables and version, the policy drawing under sampling: a run’s rollout built before
the run draws its weights. Its version before then is -1, behind every update.
NativeRolloutServer.submit
Section titled “NativeRolloutServer.submit”def submit( prompt: Sequence[int], max_new_tokens: int, *, key: int | jax.Array,) -> Future[Draw]Queue one draw and return its future.
A request the server refuses raises here, and the other requests keep running.
NativeRolloutServer.load
Section titled “NativeRolloutServer.load”def load(variables: Variables, version: int) -> NoneNativeRolloutServer.waited
Section titled “NativeRolloutServer.waited”def waited(pending: Callable[[], Sequence[Future]]) -> NoneReturn once pending, read afresh as draws finish, gives no future: as each finishes where an
engine steps the server, and for a pooled server stepped with every process of its mesh, on the
calling thread, until no process waits on one. Every process calls it at the same point of its run
with the futures it waits on, none for a process that waits on nothing. Each round agrees whether
any process’s server holds work, which every process then steps (Server.step), and else waits
for its own futures’ work beside the server (a reward scored, an environment stepped) to finish one
or to submit a draw. The steps run where the run’s other programs run, in the same order on every
process: a thread of their own would enter the mesh’s collectives in another order on each, and
its agreements interleave with the run’s (ledger 4279). A step that fails on any process stops the
server on every one, each step’s outcome agreed before the next round (agreed), every request it
holds settled with the failure (Server.fail).
NativeRolloutServer.close
Section titled “NativeRolloutServer.close”def close() -> NoneStop after the step in progress; every draw still outstanding fails.
OllamaCompletion
Section titled “OllamaCompletion”class OllamaCompletion(model: str, client: OllamaClient | AsyncOllamaClient)Calls one Ollama model through an ollama.Client or ollama.AsyncClient that you pass in.
Calling it, or acall with the async client, takes a batch of text
prompts and a finite token budget and returns a Completion. Each
prompt is one generate request, seeded at the root seed key plus its
row index; pass the seed as key=, not seed=. Other request fields
follow the SDK schema, including images, format, context, raw,
template, system, think and logprobs. options takes the SDK’s
Options value or a mapping of backend options. chat and achat
support tools and tool results, and stream and astream yield the
SDK’s own GenerateResponse objects.
OllamaCompletion.stream
Section titled “OllamaCompletion.stream”def stream( prompt: str, max_new_tokens: int, *, key: int | jax.Array | None = None, **parameters: RequestField = {},) -> Iterator[OllamaResponse]OllamaCompletion.chat
Section titled “OllamaCompletion.chat”def chat( messages: Sequence[ChatMessage], max_new_tokens: int, *, key: int | jax.Array | None = None, stream: bool = False, **parameters: RequestField = {},) -> OllamaChat | Iterator[OllamaChat]OllamaCompletion.acall
Section titled “OllamaCompletion.acall”async def acall( prompts: str | Sequence[str], max_new_tokens: int, *, key: int | jax.Array | None = None, **parameters: RequestField = {},) -> CompletionOllamaCompletion.astream
Section titled “OllamaCompletion.astream”async def astream( prompt: str, max_new_tokens: int, *, key: int | jax.Array | None = None, **parameters: RequestField = {},) -> AsyncIterator[OllamaResponse]OllamaCompletion.achat
Section titled “OllamaCompletion.achat”async def achat( messages: Sequence[ChatMessage], max_new_tokens: int, *, key: int | jax.Array | None = None, stream: bool = False, **parameters: RequestField = {},) -> OllamaChat | AsyncIterator[OllamaChat]OpenAICompletion
Section titled “OpenAICompletion”class OpenAICompletion( model: str, client: OpenAI | AsyncOpenAI, provider: Literal['openai', 'vllm', 'sglang'] = 'openai',)Calls one model through an OpenAI client, including one pointed at a vLLM or SGLang base_url.
provider (openai, vllm or sglang) names the engine behind the
client. The SDK’s own request options pass through to the completions
and chat resources. Engine-only controls that vLLM and SGLang both
accept, such as top_k, min_p and stop_token_ids, go in
extra_body. A Sampling passed as sampling= becomes request fields,
and its top-k, min-p, repetition-penalty and EOS controls need
provider="vllm" or "sglang". SDK responses and streaming chunks keep
the backend’s logprobs, token ids and extensions, tool calls and
structured output fields unchanged. Completion prompts are text or rows
of token ids, and a row of ids reaches the engine as ids, without being
detokenized and tokenized again.
OpenAICompletion.stream
Section titled “OpenAICompletion.stream”def stream( prompts: str | Sequence[str] | TokenRows, max_new_tokens: int, *, key: int | jax.Array | None = None, **parameters: RequestField = {},) -> Stream[OpenAIResponse]OpenAICompletion.chat
Section titled “OpenAICompletion.chat”def chat( messages: Sequence[ChatMessage], max_new_tokens: int, *, key: int | jax.Array | None = None, stream: bool = False, **parameters: RequestField = {},) -> ChatCompletion | Stream[ChatCompletionChunk]OpenAICompletion.acall
Section titled “OpenAICompletion.acall”async def acall( prompts: str | Sequence[str] | TokenRows, max_new_tokens: int, *, key: int | jax.Array | None = None, **parameters: RequestField = {},) -> CompletionOpenAICompletion.astream
Section titled “OpenAICompletion.astream”async def astream( prompts: str | Sequence[str] | TokenRows, max_new_tokens: int, *, key: int | jax.Array | None = None, **parameters: RequestField = {},) -> AsyncStream[OpenAIResponse]OpenAICompletion.achat
Section titled “OpenAICompletion.achat”async def achat( messages: Sequence[ChatMessage], max_new_tokens: int, *, key: int | jax.Array | None = None, stream: bool = False, **parameters: RequestField = {},) -> ChatCompletion | AsyncStream[ChatCompletionChunk]OpenAIRolloutServer
Section titled “OpenAIRolloutServer”class OpenAIRolloutServer( completion: OpenAICompletion, sampling: Sampling, weights: WeightSync, *, version: int = 0, workers: int = 64, processed_logprobs: bool = False, routing: bool = False, decode: Callable[[Sequence[int]], str] | None = None,)Serves rollouts from the OpenAI-compatible completions endpoint of a vLLM or SGLang engine.
Each submission is one completion request with token ids as the prompt.
It includes the Sampling policy and a seed, and asks for one
log-probability per sampled token and for the sampled ids themselves.
Up to workers requests are in flight at once. The engine is
completion.provider, which must be vllm or sglang.
The reported log-probabilities are behavior likelihoods only for some
policies. vLLM reports raw ones unless started with
--logprobs-mode processed_logprobs; pass processed_logprobs=True
when it is. vLLM returns a filtering policy’s kept ids only on its token
route, so use VLLMGenerateServer for a filtering policy. SGLang’s
/v1/completions reports the temperature-scaled distribution before its
filters, so with SGLang any temperature works but no filter does, and
SGLANG_RETURN_ORIGINAL_LOGPROB must stay unset. An SGLang draw streams,
drawn as the policy without its EOS ids and cut at the first of them
(_streamed). Both engines end a draw on the policy’s stop strings
themselves, matched on the text they decode, and return its ids through
the one completing the string, but vLLM keeps a speculative block’s ids
past it (vllm/v1/engine/detokenizer.py:118-131 at v0.30.0). So a policy
with stop strings takes decode, the policy’s tokenizer’s, and its draws
are cut at the first id whose text completes one (through_stop). The
constructor raises
ValueError for the combinations these rules exclude, and for a
temperature of zero. SGLang honors a seed only under
--enable-deterministic-inference.
With routing=True, every draw records vLLM’s routed experts for
routing replay (dew.nn.protocols.Routes); the engine must run with
--enable-return-routed-experts.
Publication
Section titled “Publication”class Publication( weights: WeightSync, *, version: int = 0, stamp: Callable[[int], None] | None = None,)Publishes weights to an engine fleet under a version, for use as a scheduler’s Publisher.
load(variables, version) pushes the weights, stamps the version, then
updates version. weights does the push (SafetensorsReload, or any
WeightSync). stamp, when set, labels later calls with a version, as
a recording gateway does (dew.interop.gateway.Gateway.stamp). The stamp
runs only after every replica serves the new version, because a gateway
stamps a call when it arrives, and a stamp ahead of a replica would claim
weights the call was not sampled from. A failed push or stamp raises and
leaves version unchanged. The launch version is stamped when the
version is first read, which a scheduler does before it submits a draw
under it, so a gateway left at a higher version by an earlier run cannot
mislabel this run’s first calls, and constructing the publication touches
no gateway. Every process calls load, and process 0 stamps.
Publication.load
Section titled “Publication.load”def load(variables: Variables, version: int) -> NoneRolloutServer
Section titled “RolloutServer”class RolloutServer(Protocol)Samples continuations of token prompts under a versioned policy whose weights can be reloaded.
submit returns a future without waiting for generation. load
replaces the served weights and sets version, and draws submitted
after it report that version.
RolloutServer.submit
Section titled “RolloutServer.submit”def submit( prompt: Sequence[int], max_new_tokens: int, *, key: int | jax.Array,) -> Future[Draw]RolloutServer.load
Section titled “RolloutServer.load”def load(variables: Variables, version: int) -> NoneRolloutServer.close
Section titled “RolloutServer.close”def close() -> NoneRunProcessor
Section titled “RunProcessor”class RunProcessor(tokenizer: Tokenizer)Adapts a run’s tokenizer to the Processor interface a task uses.
Calling it encodes each text prompt and left-pads the rows into
ModelInputs; a text run takes no images or audio. decode returns one
string per token row.
bos_id: int | None-
The id the run’s tokenizer starts a sequence with, or None.
RunProcessor.decode
Section titled “RunProcessor.decode”def decode(tokens: jax.typing.ArrayLike) -> list[str]RunProcessor.encode
Section titled “RunProcessor.encode”def encode(text: str, *, specials: bool) -> list[int]text’s ids from the run’s tokenizer, with the special tokens it adds around a sequence
where specials, and none otherwise.
RunProcessor.spelling
Section titled “RunProcessor.spelling”def spelling(ids: Sequence[int]) -> strThe text the run’s tokenizer decodes ids to, its special tokens spelled out.
SafetensorsReload
Section titled “SafetensorsReload”class SafetensorsReload( source: Pretrained, directory: Path, engines: tuple[str, ...], engine: Literal['vllm', 'sglang'], dtype: str = 'bfloat16', timeout: float = 600.0,)Publishes a policy version to engine replicas through safetensors on disk.
source.save writes the weights, the derived config and the tokenizer
files into directory, the directory every replica was launched on (a
shared one when replicas span hosts). Floating-point leaves are cast to
dtype first. The source’s writers replace each file whole (dew._files.replacing) and
publish a sharded index last, so no engine reads a half-written file. engines are the
replicas’ root URLs, not their /v1 API URLs, and timeout bounds each
HTTP call in seconds. The push fails if a call answers with any status
but 200, or, for a call that reports its outcome, with a success that
is not true. Both engines report some failures as a 200 with
{"success": false}.
For vLLM (engine="vllm", v0.30.0, VLLM_SERVER_DEV_MODE=1), the push
reloads the directory through /collective_rpc inside the publication
lifecycle it shares with NCCLPush: every replica pauses with
mode=wait, takes the weights, resets its prefix cache, sets the weight
version and resumes, and a phase that fails on any replica stops the
push before the next phase, leaving the paused replicas paused until a
later push succeeds. In-flight draws therefore finish entirely on
the old weights, and no cached prefix outlives them. For SGLang
(engine="sglang", v0.5.20), the push is one /update_weights_from_disk
call, which waits for in-flight requests, holds new ones and flushes the
radix cache. A failed SGLang load rolls back by re-reading the same
directory.
A failed push raises an error that names every replica that did not take
the version. Every process of a pool calls the push: each runs the same
writer over the served tree and every other tree the source writes beside
it (a decision release’s readout, _published), reading each tensor home
together as it is written (reading_home), and process 0 alone writes the
weights and publishes them; an agreement point raises a failure on every
process, so the others do not hang at the next collective.
SafetensorsReload.write
Section titled “SafetensorsReload.write”def write(variables: Variables) -> NoneWrite variables into directory, replacing each file atomically.
Every process of a pool calls it.
Scoring
Section titled “Scoring”class Scoring(Protocol)A task served by forward passes over its requests’ rows (ScoringServer). It holds no weights
of its own: the server holds them resident, and hands them to every pass, so a task and the weights
it scores with are one pair the server swaps whole (ScoringServer.reload).
Scoring.admit
Section titled “Scoring.admit”def admit(request: Request) -> ScoringRequest[Row, Payload]request laid out as the rows the task’s passes read; raises ValueError for a request it
refuses.
Scoring.planned
Section titled “Scoring.planned”def planned() -> PassPlan[Row]An empty pass of the task’s, which takes rows one at a time while they are one pass: within its budget, and alike enough for one compiled forward.
Scoring.forward
Section titled “Scoring.forward”def forward(variables: Variables, rows: Sequence[Row]) -> Sequence[Output]One pass over rows with variables: each row’s output, in order.
Scoring.compile
Section titled “Scoring.compile”def compile(variables: Variables) -> Sequence[object]Compile every program the task’s passes run with variables, up front, so a server
compiles none while it serves; returns the shapes it serves at, one program each.
Scoring.answer
Section titled “Scoring.answer”def answer(payload: Payload, outputs: Sequence[Output]) -> AnswerA request’s answer from what it admitted and its rows’ outputs, in its rows’ order.
ScoringRequest
Section titled “ScoringRequest”class ScoringRequest(rows: tuple[Row, ...], payload: Payload)A request as its task admitted it: its rows, each scored in one of the task’s passes, and what the task’s answer reads of it besides their outputs.
ScoringServer
Section titled “ScoringServer”class ScoringServer( task: Scoring[Request, Row, Payload, Output, Answer], variables: Variables, *, rows: int | None = None,)Answers scoring requests through one task’s passes, one pass a step.
A pass takes the waiting rows oldest first, as many as the task’s plan takes (planned), all
laid out by one task: a request admitted before a reload is scored by the task and weights that
admitted it.
A step looks at every waiting request while the pass’s plan takes their rows, or at most rows,
cancelled ones among them: a pass that looks no further than a few requests leaves the windows of a
larger shape part empty: over a set of 17.5M real tokens, passes of 64 requests at most computed
22.3M tokens in 389 passes, passes over every waiting request 20.1M in 308, the same programs.
Build one with from_task.
arrived: Callable[[], None]-
Called after each submission: an
Engineserving this kind wakes to it. pooled-
Whether its weights’ mesh spans several processes (
Kind.pooled), which anEngine, stepping a kind of one process, refuses. programs: tuple[object, ...]-
The shapes of the programs compiled up front (
from_task), one each. busy: bool-
Whether a request waits for a pass, or for the rest of its rows.
ScoringServer.from_task
Section titled “ScoringServer.from_task”def from_task( task: Scoring[Request, Row, Payload, Output, Answer], variables: Variables, *, rows: int | None = None,) -> ScoringServer[Request, Row, Payload, Output, Answer]A server over task’s passes with variables, held resident, every program its passes run
compiled up front (Scoring.compile, the shapes in programs); a step looks at every waiting
request, or at most rows. A server built directly compiles each program on its first pass.
ScoringServer.metrics
Section titled “ScoringServer.metrics”def metrics() -> dict[str, float]What the task it serves reports of its passes so far (Metered), the share of their tokens its
rows held among them; none for a task that reports nothing.
ScoringServer.submit
Section titled “ScoringServer.submit”def submit(request: Request) -> Timed[Answer]Queue one request and return its future answer. The task lays it out here, so a request it refuses raises its ValueError here, and the others go on.
ScoringServer.step
Section titled “ScoringServer.step”def step() -> NoneRun one pass over the waiting rows, and answer every request whose last row it scores.
ScoringServer.run
Section titled “ScoringServer.run”def run() -> NoneStep until no request waits.
ScoringServer.reload
Section titled “ScoringServer.reload”def reload( task: Scoring[Request, Row, Payload, Output, Answer], variables: Variables,) -> NoneServe task with variables in place of the current pair, both at once, for every request
submitted after: a request is scored by the pair that laid it out, so its answer pairs one
task with its own weights, as a task’s calibration is fitted to the weights it came with.
The tree must match the served one (dew.inference.engine.normalized), and its leaves are
copied onto the served placement before the swap. Safe against a step on another thread.
ScoringServer.fail
Section titled “ScoringServer.fail”def fail( failure: BaseException, taken: Sequence[_Request[Row, Payload, Output, Answer]] = (),) -> NoneStop serving after failure: every waiting request, and taken’s, settles with it, and a
later submission or step is refused.
Server
Section titled “Server”class Server( model: nn.Module, variables: Variables, processor: Processor | None, *, sampling: Sampling, transforms: tuple[LogitsTransform, ...], stopping: Finishing, grammar: Grammar | None, rows: Rows, slots: int, capacity: int, admission: int, default_budget: int | None, max_length: int | None, decode_steps: int, prefix: int | Literal['prompt'] | None = None, precision: Numerics | None = None, progress: Progress = END, prefill_mesh: Mesh | None = None,)Serves text requests with continuous batching over one resident KV cache.
A TextGeneration call runs its batch as one program, so no request can
join until it returns. A server keeps slots rows of capacity cache
slots each and refills them from a queue. Each step prefills the prompts
admitted that step into free rows and draws one token for every occupied
row, or a speculative block where the task speculates (from_task). A
row leaves where the task’s criteria end it (EOS, a stop string) or
at its budget, and the next request takes its slot. Every request runs
the task’s policy and draws the same tokens it would draw through the
task alone with the same key, except where the model’s products round at
boundaries that the two programs’ roundings cross (the module docstring).
Build one with from_task. admission bounds the prompts one step
prefills. A request whose prompt and budget do not fit the capacity is
refused at submit, with the error TextGeneration raises for a request
over the model’s context. Over a mesh, the slots and the admission split
evenly over the groups of rows, and so do the pages of a paged pool. The
server holds the weights at their decoding storage
(dew.interop.weights.decoding_storage), to which reload casts the
weights it is handed.
cross: tuple[CrossBlocks, int, np.dtype] | None-
The model’s cross-attention layers and its context’s width and dtype, where a request may carry a context beside its prompt.
arrived: Callable[[], None]-
Called after each submission: an
Engineserving this kind wakes to it (dew.inference.engine.Kind). prefix_hits-
The number of prompt tokens served from shared prefix pages instead of prefilled.
pooled-
Whether the server’s mesh spans several processes, which step it together (
run,step); anEnginesteps a server of one process. group_fields-
The group fields some layer reads whole (
dew.nn.kv_cache.GROUPS), whose groups its cache keeps beside the keys: the ones a served prompt’s slot beside its tokens. keyed-
Whether a layer’s decode cache holds its rows’ rounding keys (
dew.nn.kv_cache.ROUNDING_KEYS), so a prompt’s state follows its request’s key, which then names its prefix too (_prepared). mixed_refusal-
Why the admitting step prefills in a separate forward, or None when it runs one mixed forward.
The mixed forward covers each row’s last draw and the admitted prompts in one pass, so the projections read their weights once.
cache: Variables-
The resident cache, updated in place by every step.
occupancy: int-
The number of rows the host knows to be running, one step behind the device.
busy: bool-
Whether a request waits or runs, or a call’s draws wait to be read.
Server.from_task
Section titled “Server.from_task”def from_task( task: TextGeneration, *, slots: int, capacity: int, admission: int | None = None, kv_cache: KVCache | None = None, chunk: int | None = None, prefix_cache: bool = False, prefix_states: int | None = None, prefix_state_every: int | None = None, host_pages: int = 0, decode_steps: int = 1, prefill_mesh: Mesh | None = None,) -> ServerBuild a server over the task’s model, weights, processor and policy.
capacity is rounded up to whole 64-slot tiles and whole pages, and
must stay within the model’s context. The task’s n must be one, and
its strategy the row-wise sampler, with or without a grammar, or
speculative blocks (Speculative): each iteration then drafts,
verifies and emits a block in every row, which draws as the row-wise
sampler draws, over a dense cache whose prompts prefill whole.
kv_cache replaces the model’s cache layout (dew.nn.kv_cache). A
paged layout pools the pages of every row, so short requests fit more
rows than pages * page_size / capacity. chunk prefills a prompt in
pieces of at most that many tokens, one piece a step, so a long prompt
does not stall the rows beside it. It also works over a dense cache
where the admitting step is mixed. prefix_cache lets a request reuse
the pages of a prompt prefix an earlier request computed, and needs a
paged cache. Where the model keeps a row’s own state beside its pages
(a recurrent mixer’s memory, latent attention’s index keys), a reused
prefix is read with the state its prompt left at its end: each group
keeps prefix_states of them (one a slot by default), taken at a
prompt’s deepest shareable page boundary and every
prefix_state_every pages below it, a piece cut to end there, and a
request reuses pages as deep as a state is kept
(dew.inference.pages.States). host_pages keeps that many more
prefix pages in host memory, split over the groups: a cached page the
pool reclaims goes to the host first, and a prompt that shares it reads
it back into a fresh page before its prefill (dew.inference.pages), so
a prefix cache holds more than the device’s pages. Where rows keep a
state beside their pages, a state giving way goes to the host too, as
many states as the bytes of host_pages pages hold.
prefill_mesh prefills the admitted prompts on a mesh of their own,
as MaxEngine’s and vLLM’s disaggregated serving does: the server keeps
a copy of its weights there, placed as on its own mesh, prefills each
admission’s rows over it and moves their decoder state to the decoding
mesh, where the rows are seated and decode; a request draws what it
draws on one mesh. The rows prefill whole over a dense cache.
Weights on a mesh are served on that mesh. The slots, the admission
and the pages are split over its row axes, so the number of row
groups must divide the slots and the admission. The admission
defaults to the largest multiple of the group count up to eight rows
an iteration. The model decodes its stack and sequences whole on a
mesh with a stage or sequence axis (dew.nn.sharding.whole). On a
mesh over several processes each process seats and reads its own
groups’ rows and the pool steps the server together (run).
decode_steps runs that many iterations per device call, so the host
launches the step’s kernels once per call; on a tensor axis those
launches take about as long as the step. Requests are seated, and
draws read, only at call boundaries, so a request waits up to
decode_steps iterations for its slot. The default admission seats
that many iterations’ rows per call. The draws are the same for any
value. More iterations pay off where the host’s launches leave the
devices idle, as across an NVLink pair at 128 slots
(docs/concepts/inference.md). On one RTX 4080, which was already
busy, more iterations were slower at 32 and 128 slots, and at 64
slots only two iterations gained, by under 1%. A call seats requests
only before its first iteration, so the slots take more iterations
to fill (docs/performance.md).
A server draws at the progress its task draws at (TextGeneration.progress), the run’s end or a
run’s, and reload moves it to the version it serves.
Server.reload
Section titled “Server.reload”def reload(variables: Variables, progress: Progress | None = None) -> NoneServe variables from the next step on, in place of the current weights, at the run’s
progress where one is given (a run’s version, NativeRolloutServer.load), else at the one it
served. A progress takes the place of one of its kind: a server built at the run’s end serves
that end, and one built at a run’s progress serves a run’s progress, its steps compiled for it.
The tree must match the served one leaf for leaf in shape, and a leaf
that is not floating point must match its dtype too. A floating leaf
is cast to the served precision, so the compiled step keeps running
without a recompile. The leaves are copied onto the served placement,
so the caller may donate its buffers right after. Running rows keep
their cache and draw their next token from the new weights, and
prefix pages the old weights wrote are no longer shared. reload is
not thread-safe against step, so the caller must serialize the two.
A server drawing speculative blocks reloads between requests alone: a
running row’s next block opens on the boundary logits its last block
verified under the old weights, which no next-step draw recomputes.
Server.submit
Section titled “Server.submit”def submit( prompt: Prompt, max_new_tokens: int | None = None, *, key: int | jax.Array, adapter: str | None = None, shared: int | None = None,) -> TicketQueue one request and return a ticket that resolves to its Generation.
prompt is one text prompt, ModelInputs row or token row, without
media or noncanonical positions; a ModelInputs row may carry the
context a cross-attention decoder reads (dew.nn.inputs.CONTEXT) and
the image and block groups its layers read whole
(dew.nn.kv_cache.GROUPS), which slot beside its tokens. It is
validated as TextGeneration validates it, against the server’s
capacity in place of the model’s context.
Without max_new_tokens, the task’s default budget applies, its
max_new_tokens or what its max_length leaves past the prompt. A
request with a zero budget resolves at once, with the prompt alone.
Another thread may submit while one steps. The request is validated
here and queued at once; a step seats it when a slot is free.
adapter names the request’s adapter where the task’s model is an
adapter set’s (dew.lora.AdapterSet), each row drawing under its own:
a request draws what a task over that adapter alone draws, and the
base model’s where it names none.
shared declares the prompt’s first shared tokens a segment other
requests share, wherever it ends (a decision’s encoded state, each
question past it a request of its own): a paged server with
prefix_cache keeps the keys and the row state at its end, and a
request whose prompt starts with the same segment, declaring it too,
prefills only its rest, the page the segment ends inside copied into
its own (dew.inference.pages.segment_hash). Such requests submitted
together wait for the first to write it. A segment ends between two
tokens no group (a block, an image) holds both of.
Server.step
Section titled “Server.step”def step() -> NoneRun one device call and read the previous call’s draws.
The call admits the queued requests that fit and runs decode_steps
iterations. Its own draws are read by the next step, or by run.
Server.run
Section titled “Server.run”def run() -> NoneStep until every queued and running request has resolved. A step that fails stops the server:
every request it holds settles with the failure (fail), which run raises.
Server.fail
Section titled “Server.fail”def fail(failure: BaseException) -> NoneStop after failure: every waiting and running request settles with it, and a later
submission or step is refused.
TextToImage
Section titled “TextToImage”class TextToImage( model: nn.Module, process: Process, inputs: InputSpec, variables: Variables, autoencoder: AutoEncoder | None = None, steps: int = 50, guidance: Guidance | None = None, solver: Solver[object] = DDIM(), grid: Callable[[int], tuple[Process, jax.Array]] | None = None, final_denoise: bool = True, finish: Callable[[Variables, jax.Array], jax.Array] | None = None, *, precision: Numerics | None = None,)Generates images from text prompts with a trained diffusion model and its encoders.
Call it as pipe(prompts, key=0) or
pipe(prompts, steps=40, guidance=4.0, solver=Heun(), key=key).
variables is the objective’s whole tree, with the EMA copy merged over
the live weights when the run kept one, so a sample comes from the weights
the run publishes. steps, guidance and solver are the defaults for a
call that omits them; an objective or a loaded source sets them. grid is
for a source whose solver pairs its own sigma and model-time tables, and
returns the process and its explicit time grid for a step count. With
final_denoise False a trajectory ends the way those solvers end it.
finish runs on the decoded images under the same placement, for a source
that ships a checker or an output transform.
The weights stay where they are placed. On a mesh, the prompts are split into per-process rows over its batch axes, and the result keeps that sharding. Each row’s initial noise comes from its global row index, so a pool of processes draws the same noise as one process does for the same prompts.
latent_shape: tuple[int, ...]-
The per-example shape the model denoises.
It is the sample field’s shape, or that field’s latent shape when the task has an autoencoder in front of the model.
TextToImage.bind
Section titled “TextToImage.bind”def bind(variables: Variables) -> TextToImageReturn this task with variables as its weights, its encoders’ included: each call
encodes its prompts and its blank with the towers it holds.
TextToImage.quantized
Section titled “TextToImage.quantized”def quantized(spec: Quantization) -> TextToImageReturn this task with its denoiser’s weights quantized as spec says.
The denoiser’s matmuls then compute with the quantized weights
(dew.quantization.quantize_for_serving). The encoders and
the autoencoder keep their weights.
TextToImage.from_objective
Section titled “TextToImage.from_objective”def from_objective(objective: DiffusionObjective, variables: Variables) -> TextToImageBuild a task over the objective’s model and variables that samples the way its evaluation does.
A loss-only head the objective trains is dropped.
TextToImage.from_saved
Section titled “TextToImage.from_saved”def from_saved( run: SavedRun, *, ema: bool | None = None, mesh: MeshSpec | Mesh | None = None, layout: Layout | None = None, dtype: DTypeLike | None = None, param_dtype: DTypeLike | None = None, variables: Variables | None = None,) -> TextToImageLoad the run run (its checkpoint pinned, SavedRun.pinned), built from the inference record
its checkpoint holds (Checkpoints.artifact) the way the recipe built it; the run’s run.json
is not read. from_run loads one from its directory.
ema None reads the averaged weights when the run kept them and the
live ones when it kept none; True requires the averaged weights, and
False reads the live ones. A run whose averaged weights are a reference
policy, such as Flow-GRPO’s frozen KL reference, always reads its live
policy. With mesh, the weights are restored directly onto that mesh
under layout, the way the trainer places them. Without it, they go
onto a default MeshSpec() over the current pool’s devices.
dtype sets the compute dtype of the model, the encoders and the VAE.
param_dtype sets the dtype the parameters are stored in; None keeps
the checkpoint’s dtypes exactly. Supplied variables, a tree of the
run’s own form (a run that started from it saved them), are bound in
place of the checkpoint’s, which is not read.
TextToImage.from_flaxdiff
Section titled “TextToImage.from_flaxdiff”def from_flaxdiff( directory: str | os.PathLike, config: Mapping[str, object], *, jax_version: str, ema: bool = True, best: bool = False, dtype: DTypeLike | None = None,) -> TextToImageLoad a FlaxDiff text-to-image run (simple_udit or hybrid_dit on the SD VAE) into Dew’s model.
directory is one checkpoint step, config the run config FlaxDiff’s
trainer logged, and jax_version the jax version the run trained
under, from its requirements.txt. ema and best pick the weights;
dtype is the model’s compute dtype. dew.interop.flaxdiff reads the
format.
TextToImage.prepared_process
Section titled “TextToImage.prepared_process”def prepared_process(steps: int) -> tuple[Process, tuple[float, ...] | None]Return the process and explicit time grid that a call with steps steps runs.
The grid is a tuple of concrete times, so the compiled trajectory has a
fixed length and fixed values. It is None when the task has no grid.
Raises ValueError unless steps is a positive int.
TextToImage.prepare
Section titled “TextToImage.prepare”def prepare( prompts: str | Sequence[str | Mapping[str, object]] | None = None, *, conditions: Mapping[str, Conditioning] | None = None, key: int | jax.Array, steps: int | None = None, unconditional: str | Sequence[str | Mapping[str, object]] | Mapping[str, Conditioning] | None = None, image: ArrayLike | None = None, image_latents: ArrayLike | None = None, mask: ArrayLike | None = None, noise: ArrayLike | None = None, initial: ArrayLike | None = None, times: ArrayLike | Sequence[float] | None = None, encode_key: int | jax.Array | None = None,) -> DenoisingInputsEncode the conditions and build the initial state for a call, on a concrete grid.
Pass either prompts or conditions. The result is DenoisingInputs,
which a call takes in place of prompts. Each array argument has one row
or one row per sample. The optional inputs:
image: uint8 or normalized floating NHWC pixels at the task’s geometry.image_latents: the image already encoded, which skips VAE encoding.mask: spatial conditioning added to both guidance branches; it needsimageand an autoencoder.noise: unit Gaussian noise for noising a clean image.initial: an already-noisy latent state for a continuation or refiner handoff, which is never noised again.times: an explicit grid that selects a partial trajectory in the prepared process.encode_key: the key that samples a VAE posterior; None uses its mean.
conditions are prompts already encoded, {keyword: condition} with
one row per sample. A pipeline’s conditioner loaded on its own can
encode them, so the text encoder does not have to share the device with
the denoiser (load_diffusion_source(text=False)). unconditional
encoded the same way, one row or one per sample, is the branch guidance
reads. It can also be negative prompts as text, which the text encoder
encodes. Without it, the task’s own blank prompt is encoded if the text
encoder is loaded. If it is not loaded, the result has no unconditional
branch, and a call with it must pass guidance=None. Each row’s noise
is the noise a call with prompts draws.
Raises ValueError for a combination the task cannot run, such as
image together with image_latents, or noise without a clean
image.
TextToImage.scoring
Section titled “TextToImage.scoring”def scoring( *, steps: int | None = None, guidance: Guidance | float | None | Omitted = OMITTED, solver: Solver | None = None, decode: bool = True, images: int = 8,) -> tuple[ImageScoring, Variables]This task as a scoring task of ImageRequests (dew.inference.scoring.Scoring), which a
ScoringServer serves beside the other kinds of an Engine, and the weights it samples with.
steps, guidance and solver are a request’s where it names none, the task’s where these are
omitted. A request is answered with the Images its lone call task(prompts, key=key, ...) draws
with its settings: its noise and its walk’s draws are its key’s, whatever requests share its pass.
A pass holds requests alike (as many prompts and negatives, the same settings) up to images
images, and walks them as one program mapped over the requests, each request’s walk its own. The
scoring task holds no weights: a server holds them resident and swaps both whole on a reload
(ScoringServer.reload).
class Timed()A request’s future, with when it was submitted, admitted to a step and finished, in
time.perf_counter seconds; admitted and finished are None until they happen.
class Usage( prompt_tokens: int | None = None, completion_tokens: int | None = None, total_tokens: int | None = None,)Token usage the backend reported for the whole request.
Each count is None when the backend did not report it.
VLLMGenerateServer
Section titled “VLLMGenerateServer”class VLLMGenerateServer( base_url: str, sampling: Sampling, weights: WeightSync, *, version: int = 0, workers: int = 64, routing: bool = False, timeout: float = 600.0,)Serves rollouts from vLLM’s token route, POST /inference/v1/generate.
This is the one vLLM route that returns, beside the sampled ids and
their likelihoods, the ids the sampler kept for each of them
(GenerateResponseChoice.sampling_mask). So a top-k or top-p policy
trains on its recorded support (support_log_probs). Start the engine
with:
--enable-scale-out(or--tokens-only);--return-sampling-mask, which needs Model Runner V2 and no speculative decoding;--logprobs-mode processed_logprobs, so the reported likelihoods are the filtered ones;--enable-return-routed-expertswhenroutingis set.
vLLM builds the mask only under a finite top-k, so a Sampling without
top_k raises ValueError. The route draws under temperature, top-k,
top-p and min-p alone, so a Sampling that sets a repetition, presence
or frequency penalty, no_repeat_ngram_size, min_new_tokens,
typical_p or stop raises ValueError too.
pipeline
Section titled “pipeline”def pipeline( source: str, *, mesh: MeshSpec | Mesh | None = None, layout: Layout | None = None, dtype: DTypeLike | None = None, param_dtype: DTypeLike | Literal['auto'] | None = None, ema: bool | None = None, step: int | str | None = None, revision: str | None = None, trust: Sequence[str] = (), options: SourceOptions | None = None, precision: Numerics | str | None = None,) -> TextToImage | TextGeneration | BlockGeneration | MaskedGeneration | SavedTaskLoad the inference task for source, with its weights placed once.
source is a run directory, a source checkpoint directory or a Hub
repository holding either. A run loads as the task its recorded objective declares
(Objective.saved_task), including a plugin objective’s own task.
mesh places the weights on that mesh under layout, or under the
trainer’s default layout when layout is None. Without mesh, data
parallelism uses the current pool’s devices.
dtype sets the computation dtype, given as a dtype (jnp.bfloat16) or
its name. param_dtype sets parameter storage. None and 'auto' keep
each tensor’s stored dtype, a run’s or a source’s (a quantized source’s
packed weights decode to the dtype it declares); jnp.float32 gives a
source FP32 masters. ema
selects a run’s averaged weights: None reads them when the run kept them
and its live weights otherwise, and True always reads them. step
selects a run’s checkpoint and revision pins a Hub source; passing
revision for a run directory or step for a source raises
ValueError. trust names the packages outside Dew a run’s record may
import, and the sources whose own code a load may run (a media processor’s,
LFM3-VL’s), as trust_remote_code does in transformers. options selects
what a source builds, as a run’s --source options do (SourceOptions:
a wrapper’s language model alone, the prediction depths kept, each layer
kind’s attention); a run’s record selects its own, so passing them for a
run directory raises ValueError. precision gives a source’s task its
numerics (Pretrained.load(precision=...): its dtypes, and the precision and
forms its calls trace under); a run’s record names its own, so passing it for
a run directory raises ValueError.
Loading a task also points XLA at the on-disk executable cache, unless a cache directory is already set, so a restarted process reuses what it already compiled.