dew.objectives.rl.harbor

Harbor trials as a session source, each model call recorded by rllm-model-gateway.

HarborSource runs one Harbor trial per sample: Harbor starts the task’s sandbox, installs and runs the harness (mini-swe-agent by default), then runs the task’s verifier. The harness reaches the model only through a recording gateway, rllm-model-gateway, at a base URL that carries the session in its path, /sessions/{session}/v1. The gateway forwards each call to a vLLM or SGLang replica, asks it for token ids and log-probabilities, and records them under that session. Attribution rides in the URL, not in a header a CLI harness may drop (memo section 5.7).

Harbor and the gateway are separate services in their own environments: Dew starts harbor trials start as a subprocess, reads the trial’s result.json, and reads the session’s traces over HTTP. Nothing of either is imported here. The gateway must persist traces before it answers (sync_traces: true), so a session’s traces are complete once its harness has exited.

calls maps gateway traces onto Call records: the engine’s prompt ids, sampled ids and behavior log-probabilities as recorded, in submission order, with the version the gateway stamped when the request arrived (the publication stamps it, see Publication). The gateway extracts vLLM’s ids and SGLang’s prompt ids; SGLang’s sampled ids (choices[0].response_token_ids) are read from the raw response it keeps.

outcome decides how a trial ended from Harbor’s result and the calls:

• the verifier scored a trial whose harness exited cleanly: COMPLETED;
• the harness exited nonzero and the verifier still scored it: AGENT_ERROR with that score;
• the agent ran out of time, context or output budget, the harness hit its own step limit, or the last call stopped at its length limit: TRUNCATED, with the verifier’s reward when it ran; the scheduler’s truncation policy decides whether it trains;
• anything else: a sandbox, gateway, engine or verifier failure, a trace without ids or likelihoods, an aborted call, or a session with no call: INFRA_ERROR, which the scheduler retries and never trains on.

Name	Summary
HARBOR_KEY	Task.data[HARBOR_KEY] is the Harbor task directory the trial runs.
Gateway	The parts of an rllm-model-gateway a session source reads.
Recorded	One session as the gateway recorded it: its model calls, and the engine errors it answered instead.
calls	The gateway’s traces of one session as Calls in submission order, and its engine errors.
outcome	Status, reward, reward components and failure detail of one finished trial.
HarborSource	A SessionSource that runs each sample as one Harbor trial behind a recording gateway.

HARBOR_KEY

attribute source

HARBOR_KEY = 'harbor'

Task.data[HARBOR_KEY] is the Harbor task directory the trial runs.

Gateway

class source

class Gateway(
    url: str,
    *,
    sandbox_url: str | None = None,
    client: httpx.Client | None = None,
)

The parts of an rllm-model-gateway a session source reads.

url is the gateway’s root as Dew reaches it; sandbox_url is where the sandboxes reach it, which is what a harness’s base URL is built from.

rllm-model-gateway (3b40c37) has no authentication, and one port serves the model proxy beside GET /sessions, every session’s traces, POST /traces/query, session deletes, POST /admin/workers and POST /admin/weight_version. A sandbox runs the policy’s own commands, so a policy that reaches that port can read its group members’ transcripts, reset the version stamp the staleness bound reads, or register a worker that returns fabricated ids and likelihoods. Therefore:

• url is on an interface only Dew reaches (bind the gateway to loopback or a private trainer network);
• sandbox_url is a reverse proxy in front of it that forwards only POST /sessions/<session>/v1/chat/completions (nginx location ~ ^/sessions/[^/]+/v1/chat/completions$) and answers 403 to everything else, on its own address;
• the task’s agent phase allows that address and nothing else: Harbor’s [agent] network_mode = "allowlist" with allowed_hosts naming the proxy, or --allow-agent-host, on a provider that supports allowlists (Harbor’s tasks/network-policy page; Docker needs nftables fib support), and [verifier] network_mode = "no-network". Harbor’s allowlist filters by host, not path, which is why the proxy is needed.

sandbox_url defaults to url only for trusted harnesses and tests.

Gateway.session

def session(session: str) -> str

The OpenAI base URL a harness uses so its calls are recorded under session.

Gateway.traces

def traces(session: str) -> Sequence[object]

Gateway.ready

def ready(timeout: float, *, poll: float = 2.0) -> None

Wait until the gateway routes to a healthy worker that answers, or raise after timeout seconds.

Ready means /health/workers counts a healthy worker and GET /v1/models, proxied through the gateway to an engine, answers 200.

rllm-model-gateway marks a worker dead after three failed health checks, as happens to an engine still loading when the gateway starts, and until a later check revives it every proxied call answers a plain-text 500 that leaves no trace. A session submitted then would fail for the gateway’s reasons, not the policy’s.

Gateway.stamp

def stamp(version: int) -> None

Make the gateway label the calls it records from now on with version (a Publication stamp).

Gateway.forget

def forget(session: str) -> None

Recorded

dataclass source

class Recorded(calls: tuple[Call, ...], errors: tuple[str, ...])

One session as the gateway recorded it: its model calls, and the engine errors it answered instead.

calls

function source

def calls(traces: Sequence[object], *, unstamped: int) -> Recorded

The gateway’s traces of one session as Calls in submission order, and its engine errors.

A trace records its arrival indirectly: timestamp is when the answer was stored and latency_ms how long the engine took, so the calls are ordered by their difference. A trace without a version stamp, from a gateway no publication has stamped, takes unstamped, the version the session was submitted under: no later push can have served it anything older. The gateway also records an engine’s error reply (a prompt past the context length, an engine fault): such a trace carries error in its raw response and is an event of the session, returned as its message, not a call. A successful reply without ids, with one likelihood too few or too many, or with a field of the wrong JSON type raises ValueError; training on it would mean guessing or re-tokenizing text.

outcome

function source

def outcome(
    trial: JSON,
    records: tuple[Call, ...],
    *,
    errors: Sequence[str] = (),
    harness_exit: str | None = None,
) -> tuple[Status, float | None, dict[str, float], str]

Status, reward, reward components and failure detail of one finished trial.

trial is Harbor’s TrialResult as JSON, records the session’s calls, errors the engine errors the gateway recorded for it, and harness_exit the harness’s own exit status when it reports one. An engine that refused an overflowing prompt truncated the rollout; any other engine error makes it infra, even when the harness retried and went on. A result whose fields have the wrong JSON types is infra: nothing in it can be trusted to score.

HarborSource

class source

class HarborSource(
    gateway: Gateway,
    *,
    harbor: str | os.PathLike[str],
    model: str,
    trials: os.PathLike[str],
    agent: str = 'mini-swe-agent',
    environment: Mapping[str, str] | None = None,
    arguments: Sequence[str] = (),
    workers: int = 8,
    grace: float = 60.0,
    ready_timeout: float = 900.0,
    ready_poll: float = 2.0,
)

A SessionSource that runs each sample as one Harbor trial behind a recording gateway.

harbor is the Harbor executable (in its own environment), agent and model its --agent and --model, and trials the directory trials are written under. environment is passed to the harness as agent environment (--ae), beside the per-trial OPENAI_BASE_URL that carries the session; arguments are further harbor trials start options (environment provider, timeouts, agent kwargs). workers trials run at once; Harbor’s own sandbox limits apply inside each. The harness must speak the OpenAI chat API through OPENAI_BASE_URL, as Harbor’s mini-swe-agent does.

Gateway sessions are named {task}:{group}:{sample}:{token}, where group is unique to one submit across runs and token is 128 secret bits, so a sandbox can address only the session it was handed. Every future resolves to a Session: a failure of Harbor, the sandbox, the gateway or the engine is an INFRA_ERROR session, not an exception, and a cancelled trial is a CANCELLED one. attempt is always 0: a retry is a fresh submit, relabelled by the scheduler that owns group identity.

The first submit waits, up to ready_timeout seconds, until the gateway reports a healthy worker (Gateway.ready), so a source started beside engines that are still loading launches no trial the gateway would answer with a traceless 500.

Use it as a context manager, or call close, to cancel every trial on the way out. A trainer that exits without either (an uncaught KeyboardInterrupt) still stops its trials: trials run on daemon threads, and an atexit hook cancels queued trials and interrupts, then kills after grace, the running ones.

HarborSource.submit

def submit(task: Task, samples: int, *, version: int) -> list[Future[Session]]

HarborSource.cancel

def cancel(futures: Sequence[Future[Session]]) -> None

Interrupt the named trials; Harbor tears their sandboxes down, and each resolves CANCELLED.

HarborSource.close

def close() -> None

Cancel every unresolved trial, queued or running, and wait for the running ones to tear down.