Skip to content

dew.interop.gateway

The seam between an agent harness and the policy: rllm-model-gateway, recording every model call.

A harness reaches the model only through a recording gateway, rllm-model-gateway, at a base URL that names the session in its path, /sessions/{session}/v1. The gateway forwards each call to a vLLM or SGLang replica, asks it for token ids and log-probabilities, and records them under that session. The session is part of the URL because a CLI harness may drop a header. The gateway is a separate service in its own environment; Dew reads a session’s traces over HTTP and imports none of it. It must persist traces before it answers (sync_traces: true), so a session’s traces are complete once its harness has exited.

calls turns gateway traces into Call records, in submission order. Each holds the engine’s prompt ids, sampled ids and behavior log-probabilities as recorded, and the version the gateway stamped when the request arrived (the publication sets that stamp; see Publication). The gateway extracts vLLM’s ids and SGLang’s prompt ids. SGLang’s sampled ids (choices[0].response_token_ids) are read from the raw response the gateway keeps, and each call’s Transcript from the chat messages the harness sent and the message the engine answered, which the gateway records beside the ids. engine_failure and refused_prompt read what the engine itself said of a session, which every harness’s verdict weighs the same way (dew.interop.harness).

NameSummary
SESSIONWhat a session id may hold: it goes in a URL’s path.
GatewayConnects a session source to an rllm-model-gateway.
RecordedOne session as the gateway recorded it.
callsThe gateway’s traces of one session as Calls in submission order, and its engine errors.
engine_failureWhy the engine failed the session, or None: an error reply other than a prompt past its context length, or an aborted call.
numbervalue, a JSON number named name, as a float; ValueError for anything else, a bool included.
refused_promptWhy the engine cut the session short, or None: a prompt it refused as past its context length.

attribute source

SESSION = re.compile('[A-Za-z0-9._:-]+')

What a session id may hold: it goes in a URL’s path.

class source

class Gateway(
url: str,
*,
sandbox_url: str | None = None,
client: httpx.Client | None = None,
)

Connects a session source to an rllm-model-gateway.

It builds each session’s base URL, waits for the gateway to be ready, stamps the weight version and reads a session’s traces. url is the gateway’s root as Dew reaches it. sandbox_url is the address the sandboxes reach it at, and a harness’s base URL is built from it.

rllm-model-gateway (3b40c37) has no authentication. One port serves the model proxy and also GET /sessions, every session’s traces, POST /traces/query, session deletes, POST /admin/workers and POST /admin/weight_version. A sandbox runs the policy’s own commands, so a policy that reaches that port can read its group members’ transcripts, reset the version stamp the staleness bound reads, or register a worker that returns fabricated ids and likelihoods. To prevent that:

  • Put url on an interface only Dew reaches, by binding the gateway to loopback or a private trainer network.
  • Make sandbox_url a reverse proxy in front of the gateway, on its own address, that forwards only POST /sessions/<session>/v1/chat/completions (nginx location ~ ^/sessions/[^/]+/v1/chat/completions$) and answers 403 to everything else.
  • Let the task’s agent reach that address and nothing else. Under Harbor, that is [agent] network_mode = "allowlist" and allowed_hosts naming the proxy, or --allow-agent-host; this needs a provider that supports allowlists (see Harbor’s tasks/network-policy page; Docker needs nftables fib support). Set [verifier] network_mode = "no-network" as well. Harbor’s allowlist filters by host and not by path, which is why the proxy is needed.

sandbox_url defaults to url. Leave it unset only for trusted harnesses and tests.

def session(session: str) -> str

Return the OpenAI base URL that makes the gateway record a harness’s calls under session.

Raises ValueError unless session is URL-path safe: letters, digits and ._:-.

def traces(session: str) -> Sequence[object]
def ready(timeout: float, *, poll: float = 2.0) -> None

Wait until the gateway routes to a healthy worker that answers, or raise after timeout seconds.

Ready means /health/workers and GET /v1/models, proxied through the gateway to an engine, answer 200. Only their status is read; the bodies are closed unread. The remaining deadline bounds each network operation, and a status received after it is never accepted. The check runs every poll seconds, and expiry raises RuntimeError.

rllm-model-gateway marks a worker dead after three failed health checks, which happens to an engine that is still loading when the gateway starts. Until a later check revives the worker, every proxied call gets a plain-text 500 that leaves no trace. A session submitted in that window would fail because of the gateway, not the policy.

def stamp(version: int) -> None

Make the gateway label the calls it records from now on with version (a Publication stamp).

def forget(session: str) -> None

dataclass source

class Recorded(calls: tuple[Call, ...], errors: tuple[str, ...])

One session as the gateway recorded it.

calls are its model calls in submission order. errors are the messages of the error replies the engine sent instead of a completion.

function source

def calls(traces: Sequence[object], *, unstamped: int) -> Recorded

The gateway’s traces of one session as Calls in submission order, and its engine errors.

A trace records its arrival indirectly: timestamp is when the answer was stored and latency_ms how long the engine took, so the calls are ordered by their difference. A trace without a version stamp, from a gateway no publication has stamped, takes unstamped, the version the session was submitted under: no later push can have served it anything older. The gateway also records an engine’s error reply (a prompt past the context length, an engine fault): such a trace carries error in its raw response and is an event of the session, returned as its message, not a call. A successful reply without ids, with one likelihood too few or too many, or with a field of the wrong JSON type raises ValueError; training on it would mean guessing or re-tokenizing text.

function source

def engine_failure(recorded: Recorded) -> str | None

Why the engine failed the session, or None: an error reply other than a prompt past its context length, or an aborted call. Either makes the session infra, whatever the harness did after it, since the policy did not cause it.

function source

def number(name: str, value: object) -> float

value, a JSON number named name, as a float; ValueError for anything else, a bool included.

function source

def refused_prompt(recorded: Recorded) -> str | None

Why the engine cut the session short, or None: a prompt it refused as past its context length. The policy’s own context ran out, so the session is truncated.