Skip to content

dew.interop.harness

Agent harnesses as a session source: one harness command per sample, its model calls recorded.

HarnessSource runs each sample as one trial of a Harness: a command whose agent reaches the policy only through its own gateway session (dew.interop.gateway), at the OpenAI base URL the trial is given. Dew runs the command as a subprocess in a process group of its own, reads the session’s calls back from the gateway, and asks the harness how the trial ended. The harness decides what its command is and what its artifacts say; the source owns the gateway session, the trial’s lifecycle and its cancellation, the same for every harness.

  • Harbor (dew.interop.harbor) runs a Harbor trial: Harbor starts the task’s sandbox, runs its agent harness (mini-swe-agent by default) and then its verifier.
  • Command runs any agent CLI that reads its model’s base URL from OPENAI_BASE_URL, in a working directory of its own, and scores the directory it leaves with a verifier.

How a trial ends:

  • COMPLETED when the agent exited cleanly and the verifier scored it.
  • AGENT_ERROR, with the verifier’s score, when the agent failed and the verifier still scored it.
  • TRUNCATED when the agent ran out of time, context or output budget, hit its own step limit, or its last call stopped at its length limit. It has the verifier’s reward when the verifier ran, and the scheduler’s truncation policy decides whether it trains.
  • INFRA_ERROR for anything else: a sandbox, gateway, engine or verifier failure, a trace without ids or likelihoods, an aborted call, or a session with no call. The scheduler retries these and never trains on them.

Cancelling a trial stops its harness; it does not stop a model call the engine is still decoding for it (rllm-model-gateway 3b40c37 holds a non-streaming call’s upstream connection after its client goes, rllm-org/rllm#938), so a cancelled trial’s last call may run on to its token budget.

NameSummary
CommandAny agent CLI that reaches its model through OPENAI_BASE_URL, scored on what it leaves.
HarnessOne kind of agent harness, run as one command per sample against a gateway session.
HarnessSourceRuns each sample as one trial of harness behind a recording gateway.
JudgedHow a finished trial ended: its status, the verifier’s reward and reward components, and a detail naming what happened.
TrialOne sample’s run of a harness: its command, the environment it adds, the directory it runs in (the source’s own when None) and the directory its artifacts are under.

dataclass source

class Command(
argv: Callable[[Task, Path], Sequence[str]],
verify: Callable[[Task, Path, int], float | None],
workspace: Path,
environment: Mapping[str, str] = dict(),
)

Any agent CLI that reaches its model through OPENAI_BASE_URL, scored on what it leaves.

argv(task, directory) is the command that runs task in directory, a fresh directory under workspace that is the trial’s working directory. It runs with this process’s environment plus environment and the session’s OPENAI_BASE_URL, so a command that must not see the trainer’s credentials or files runs its agent in a sandbox of its own (a container, a remote sandbox). verify(task, directory, returncode) scores the finished directory, or returns None when it cannot (the verifier itself failed), which makes the trial infra.

A trial whose command exited 0 and scored is COMPLETED, one that exited otherwise and scored is AGENT_ERROR. The engine’s own events come first: an engine failure makes it infra, and a refused prompt or a last call stopped at its length limit truncates it.

def check(task: Task) -> None
def trial(task: Task, url: str, name: str) -> Trial
def judge(
task: Task,
trial: Trial,
log: str,
returncode: int,
recorded: Recorded,
) -> Judged

class source

class Harness(Protocol)

One kind of agent harness, run as one command per sample against a gateway session.

def check(task: Task) -> None

Raise ValueError when task is not one this harness runs, before any trial starts.

def trial(task: Task, url: str, name: str) -> Trial

The command that runs task once with its model at the OpenAI base URL url, writing its artifacts under a directory named name.

def judge(
task: Task,
trial: Trial,
log: str,
returncode: int,
recorded: Recorded,
) -> Judged

How the trial ended: from what it wrote, its output log, its exit code and the calls the gateway recorded.

class source

class HarnessSource(
harness: Harness,
gateway: Gateway,
*,
workers: int = 8,
grace: float = 60.0,
trial_timeout: float | None = None,
ready_timeout: float = 900.0,
ready_poll: float = 2.0,
)

Runs each sample as one trial of harness behind a recording gateway.

  • workers is how many trials run at once.
  • trial_timeout, when set, bounds each trial in seconds from its start. A trial past this deadline is interrupted as cancel interrupts it, and resolves INFRA_ERROR, which the scheduler retries.
  • grace is how long a trial has to wind down after each signal of a cancel or a deadline (see cancel).

Gateway sessions are named {task}:{group}:{sample}:{token}. group is unique to one submit across runs, and token is 128 secret bits, so a sandbox can address only the session it was given. As a SessionSource, every future it returns resolves to a Session. A failure of the harness, its sandbox, the gateway or the engine gives an INFRA_ERROR session, not an exception, and a cancelled trial gives a CANCELLED one. Each session’s attempt is 0, because a retry is a fresh submit and the scheduler, which assigns group identity, relabels it.

The first submit waits up to ready_timeout seconds for the gateway to report a healthy worker (Gateway.ready). So a source started beside engines that are still loading does not launch trials that the gateway would answer with a 500 and no trace.

Use it as a context manager, or call close, to cancel every trial when you are done. A trainer that exits without either, for example on an uncaught KeyboardInterrupt, still stops its trials. Trials run on daemon threads, and an atexit hook cancels the queued trials and interrupts the running ones, then kills their process groups after grace seconds. The threads start, and the hook registers, at the first submit: constructing the source starts nothing.

def submit(
task: Task,
samples: int,
*,
version: int,
key: jax.Array,
) -> list[Future[Session]]
def cancel(futures: Sequence[Future[Session]]) -> None

Interrupt the given trials, each of which resolves CANCELLED; a queued one never starts.

A trial whose output is still open grace seconds after the interrupt gets SIGTERM, and SIGKILL after another grace seconds, each sent to its whole process group. Output still open a grace after the SIGKILL is held by a process that left the group, which keeps running; the trial stops reading it and resolves.

def close() -> None

Cancel every unresolved trial, queued or running, and wait for the running ones to tear down.

dataclass source

class Judged(
status: Status,
reward: float | None = None,
components: Mapping[str, float] = dict(),
detail: str = '',
)

How a finished trial ended: its status, the verifier’s reward and reward components, and a detail naming what happened.

dataclass source

class Trial(
argv: tuple[str, ...],
artifacts: Path,
environment: Mapping[str, str] = dict(),
cwd: Path | None = None,
)

One sample’s run of a harness: its command, the environment it adds, the directory it runs in (the source’s own when None) and the directory its artifacts are under.