Skip to content

dew.objectives.rl.control

PPO for continuous control in a dew.rl.env.Env: an actor-critic policy, its rollout and its loss.

The policy is Brax’s PPO policy (google/brax, brax/training/agents/ppo): a tanh-squashed normal over the action, its location and scale from one MLP (softplus(raw) + 0.001), and a separate value MLP; the loss is the clipped surrogate over advantages whitened within the minibatch, half the squared error of the value against its GAE target, and an entropy bonus, the advantages computed in the loss from the live critic over each stored sequence as Brax’s loss computes them. A step that ended its episode at a time limit bootstraps from the observation it reached; one that ended in a terminal state takes no value after it.

ControlRollout collects with the live policy and serves the collection as epochs passes of minibatches minibatches, one an update, collecting again once they are spent, as Brax’s num_updates_per_batch passes over num_minibatches do.

NameSummary
ActorCriticBrax PPO’s networks over a flat observation: the policy MLP’s location and raw scale of each of action_size action dimensions, and the value MLP’s one value.
ControlPPOObjectivePPO over an ActorCritic (the module docstring).
ControlRolloutThe trainer’s rollout (dew.training.rollout.Rollout) of a ControlPPOObjective in env, a batched Env or a Simulator: it collects unrolls sequences of unroll_length steps from each environment with the live policy, then serves them as epochs shuffled passes of minibatches minibatches, the next one each call, and collects again once it has served them all.
SavedWhat a ControlRollout carries between calls, as a checkpoint holds it (ControlRollout.saved): every key but served, which one written before the rollout counted its own lacks.
episode_returnThe mean undiscounted return of the first episode of each of env’s environments, an Env or a Simulator, over steps steps of the policy, its deterministic action where greedy: the whole episode where steps is an episode’s length (dew.rl.evaluation.episodes).
tanh_normal_entropyThe squashed normal’s entropy as Brax estimates it: the normal’s, plus the tanh’s log-Jacobian at one draw, summed over the action.
tanh_normal_log_probThe log-density of the action tanh(raw), raw drawn from N(location, scale), summed over the action: the normal’s log-density less log(1 - tanh(raw)^2), in its stable form 2 (log 2 - raw - softplus(-2 raw)).

Flax module source

class ActorCritic(
policy_layers: Sequence[int] = (32, 32, 32, 32),
value_layers: Sequence[int] = (256, 256, 256, 256, 256),
spread: float = 1.0,
)

Brax PPO’s networks over a flat observation: the policy MLP’s location and raw scale of each of action_size action dimensions, and the value MLP’s one value. Brax’s DM Control widths: four layers of 32 for the policy, five of 256 for the value. spread multiplies the policy’s scale: below 1, a policy explores less from the start (a drone’s first rollouts fly rather than tumble).

def setup() -> None
def policy(observation: jax.Array) -> tuple[jax.Array, jax.Array]

The normal’s location and scale before the tanh, each [..., action_size].

def value(observation: jax.Array) -> jax.Array

class source

class ControlPPOObjective(
model: ActorCritic,
observation_size: int,
*,
discount: float = 0.995,
lam: float = 0.95,
clip: float = 0.3,
reward_scaling: float = 1.0,
value_cost: float = 0.5,
entropy_cost: float = 0.0001,
variables: Variables | None = None,
reads: tuple[str, ...] = (),
)

PPO over an ActorCritic (the module docstring). A batch holds sequences [B, T]: each step’s observation, the pre-squash raw_action taken and its log_prob under the policy that took it, the reward, whether the step terminated or was truncated, and the following observation it reached. discount and lam are GAE’s, clip the surrogate’s, reward_scaling multiplies each reward, value_cost weighs the value loss and entropy_cost the entropy bonus. Brax’s ppo.train defaults, but the discount, Playground’s DM Control one.

def seen(observation: Observation) -> jax.Array

What the actor and critic read of an observation: its reads views joined, or it whole.

def held_variables() -> Variables | None
def fresh_variables(key: jax.Array, held: Variables | None) -> Variables
def act(
params: Params,
observation: Observation,
key: jax.Array,
memory: Memory,
) -> tuple[jax.Array, Tree, Memory]

An action for each observation, tanh of a draw from the policy, beside the draw and its log-probability (dew.rl.env.Policy).

def greedy(
params: Params,
observation: Observation,
key: jax.Array,
memory: Memory,
) -> tuple[jax.Array, Tree, Memory]

The policy’s deterministic action, tanh of its location, as Brax evaluates it.

def advantages(
values: jax.Array,
following: jax.Array,
batch: Batch,
) -> tuple[jax.Array, jax.Array]

Brax’s two passes over each sequence [B, T] (brax/training/agents/ppo/losses.py:59-98) from the critic’s values at the observations and at the following ones: the value targets, GAE’s lambda-returns, and the actor’s advantages, a TD step from each value through the next step’s target, or through the following value where the episode ended (none after a terminal state).

def model_flops(model, batch: Batch) -> float

One update’s forward work on batch, its loss’s traced program’s (traced_update_flops).

def training_flops(model, batch: Batch) -> float

One update’s work on batch, its loss’s gradient’s traced program’s (traced_update_flops).

def loss(variables: Variables, batch: Batch, step: Step) -> tuple[Ratio, Aux]

dataclass source

class ControlRollout(
objective: ControlPPOObjective,
env: Env | Simulator,
unroll_length: int = 30,
unrolls: int = 1,
epochs: int = 4,
minibatches: int = 4,
)

The trainer’s rollout (dew.training.rollout.Rollout) of a ControlPPOObjective in env, a batched Env or a Simulator: it collects unrolls sequences of unroll_length steps from each environment with the live policy, then serves them as epochs shuffled passes of minibatches minibatches, the next one each call, and collects again once it has served them all. Its count of those served is its own, so a run whose update pools several calls (accumulation), or whose optimizer holds or rejects an update, serves each collection whole, once a pass. Its carry, the environments mid episode, stays on the device between collections. metrics reports the mean return of the episodes that ended in the last collection.

def branch(name: str) -> Iterator[None]

A cooldown branch’s calls (Rollout’s branch), after which the rollout stands as the run left it: its environments mid episode, the collection it serves and their order. Nothing it carries is donated, so what the run left is kept as it is; a begun Simulator keeps its own environments (dew.rl.simulator.branched).

def saved() -> Saved | None

What the rollout carries between calls, for the run’s checkpoint (Rollout’s saved): every environment mid episode and its running return, the collection it serves its minibatches from, their order, the collection’s mean ended return and how many of them it served; None before its first call, and over a Simulator, whose environments live outside JAX and start afresh on a resume.

def saved_shapes() -> Mapping[str, object] | None

saved’s leaves’ shapes and dtypes, before any call; None over a Simulator.

def restored(tree: Saved) -> None

The rollout carrying on from a saved tree; one saved before the rollout counted the minibatches it served reads them off the updates at its next call, as the run that saved it did.

class source

class Saved(TypedDict)

What a ControlRollout carries between calls, as a checkpoint holds it (ControlRollout.saved): every key but served, which one written before the rollout counted its own lacks.

function source

def episode_return(
objective: ControlPPOObjective,
env: Env | Simulator,
params: Variables,
key: jax.Array,
steps: int,
*,
greedy: bool = True,
) -> float

The mean undiscounted return of the first episode of each of env’s environments, an Env or a Simulator, over steps steps of the policy, its deterministic action where greedy: the whole episode where steps is an episode’s length (dew.rl.evaluation.episodes).

function source

def tanh_normal_entropy(
location: jax.Array,
scale: jax.Array,
key: jax.Array,
) -> jax.Array

The squashed normal’s entropy as Brax estimates it: the normal’s, plus the tanh’s log-Jacobian at one draw, summed over the action.

function source

def tanh_normal_log_prob(
raw: jax.Array,
location: jax.Array,
scale: jax.Array,
) -> jax.Array

The log-density of the action tanh(raw), raw drawn from N(location, scale), summed over the action: the normal’s log-density less log(1 - tanh(raw)^2), in its stable form 2 (log 2 - raw - softplus(-2 raw)).