dew.objectives.rl.control
PPO for continuous control in a dew.rl.env.Env: an actor-critic policy, its rollout and its loss.
The policy is Brax’s PPO policy (google/brax, brax/training/agents/ppo): a tanh-squashed normal over
the action, its location and scale from one MLP (softplus(raw) + 0.001), and a separate value
MLP; the loss is the clipped surrogate over advantages whitened within the minibatch, half the
squared error of the value against its GAE target, and an entropy bonus, the advantages computed in
the loss from the live critic over each stored sequence as Brax’s loss computes them. A step that
ended its episode at a time limit bootstraps from the observation it reached; one that ended in a
terminal state takes no value after it.
ControlRollout collects with the live policy and serves the collection as epochs passes of
minibatches minibatches, one an update, collecting again once they are spent, as Brax’s
num_updates_per_batch passes over num_minibatches do.
| Name | Summary |
|---|---|
ActorCritic | Brax PPO’s networks over a flat observation: the policy MLP’s location and raw scale of each of action_size action dimensions, and the value MLP’s one value. |
ControlPPOObjective | PPO over an ActorCritic (the module docstring). |
ControlRollout | The trainer’s rollout (dew.training.rollout.Rollout) of a ControlPPOObjective in env, a batched Env or a Simulator: it collects unrolls sequences of unroll_length steps from each environment with the live policy, then serves them as epochs shuffled passes of minibatches minibatches, the next one each call, and collects again once it has served them all. |
Saved | What a ControlRollout carries between calls, as a checkpoint holds it (ControlRollout.saved): every key but served, which one written before the rollout counted its own lacks. |
episode_return | The mean undiscounted return of the first episode of each of env’s environments, an Env or a Simulator, over steps steps of the policy, its deterministic action where greedy: the whole episode where steps is an episode’s length (dew.rl.evaluation.episodes). |
tanh_normal_entropy | The squashed normal’s entropy as Brax estimates it: the normal’s, plus the tanh’s log-Jacobian at one draw, summed over the action. |
tanh_normal_log_prob | The log-density of the action tanh(raw), raw drawn from N(location, scale), summed over the action: the normal’s log-density less log(1 - tanh(raw)^2), in its stable form 2 (log 2 - raw - softplus(-2 raw)). |
ActorCritic
Section titled “ActorCritic”class ActorCritic( policy_layers: Sequence[int] = (32, 32, 32, 32), value_layers: Sequence[int] = (256, 256, 256, 256, 256), spread: float = 1.0,)Brax PPO’s networks over a flat observation: the policy MLP’s location and raw scale of each
of action_size action dimensions, and the value MLP’s one value. Brax’s DM Control widths:
four layers of 32 for the policy, five of 256 for the value. spread multiplies the policy’s scale:
below 1, a policy explores less from the start (a drone’s first rollouts fly rather than tumble).
ActorCritic.setup
Section titled “ActorCritic.setup”def setup() -> NoneActorCritic.policy
Section titled “ActorCritic.policy”def policy(observation: jax.Array) -> tuple[jax.Array, jax.Array]The normal’s location and scale before the tanh, each [..., action_size].
ActorCritic.value
Section titled “ActorCritic.value”def value(observation: jax.Array) -> jax.ArrayControlPPOObjective
Section titled “ControlPPOObjective”class ControlPPOObjective( model: ActorCritic, observation_size: int, *, discount: float = 0.995, lam: float = 0.95, clip: float = 0.3, reward_scaling: float = 1.0, value_cost: float = 0.5, entropy_cost: float = 0.0001, variables: Variables | None = None, reads: tuple[str, ...] = (),)PPO over an ActorCritic (the module docstring). A batch holds sequences [B, T]: each step’s
observation, the pre-squash raw_action taken and its log_prob under the policy that took
it, the reward, whether the step terminated or was truncated, and the following
observation it reached. discount and lam are GAE’s, clip the surrogate’s,
reward_scaling multiplies each reward, value_cost weighs the value loss and entropy_cost
the entropy bonus. Brax’s ppo.train defaults, but the discount, Playground’s DM Control one.
ControlPPOObjective.seen
Section titled “ControlPPOObjective.seen”def seen(observation: Observation) -> jax.ArrayWhat the actor and critic read of an observation: its reads views joined, or it whole.
ControlPPOObjective.held_variables
Section titled “ControlPPOObjective.held_variables”def held_variables() -> Variables | NoneControlPPOObjective.fresh_variables
Section titled “ControlPPOObjective.fresh_variables”def fresh_variables(key: jax.Array, held: Variables | None) -> VariablesControlPPOObjective.act
Section titled “ControlPPOObjective.act”def act( params: Params, observation: Observation, key: jax.Array, memory: Memory,) -> tuple[jax.Array, Tree, Memory]An action for each observation, tanh of a draw from the policy, beside the draw and its
log-probability (dew.rl.env.Policy).
ControlPPOObjective.greedy
Section titled “ControlPPOObjective.greedy”def greedy( params: Params, observation: Observation, key: jax.Array, memory: Memory,) -> tuple[jax.Array, Tree, Memory]The policy’s deterministic action, tanh of its location, as Brax evaluates it.
ControlPPOObjective.advantages
Section titled “ControlPPOObjective.advantages”def advantages( values: jax.Array, following: jax.Array, batch: Batch,) -> tuple[jax.Array, jax.Array]Brax’s two passes over each sequence [B, T] (brax/training/agents/ppo/losses.py:59-98)
from the critic’s values at the observations and at the following ones: the value targets,
GAE’s lambda-returns, and the actor’s advantages, a TD step from each value through the next
step’s target, or through the following value where the episode ended (none after a
terminal state).
ControlPPOObjective.model_flops
Section titled “ControlPPOObjective.model_flops”def model_flops(model, batch: Batch) -> floatOne update’s forward work on batch, its loss’s traced program’s (traced_update_flops).
ControlPPOObjective.training_flops
Section titled “ControlPPOObjective.training_flops”def training_flops(model, batch: Batch) -> floatOne update’s work on batch, its loss’s gradient’s traced program’s (traced_update_flops).
ControlPPOObjective.loss
Section titled “ControlPPOObjective.loss”def loss(variables: Variables, batch: Batch, step: Step) -> tuple[Ratio, Aux]ControlRollout
Section titled “ControlRollout”class ControlRollout( objective: ControlPPOObjective, env: Env | Simulator, unroll_length: int = 30, unrolls: int = 1, epochs: int = 4, minibatches: int = 4,)The trainer’s rollout (dew.training.rollout.Rollout) of a ControlPPOObjective in env, a
batched Env or a Simulator: it collects unrolls sequences of unroll_length steps from each
environment with the live policy, then serves them as epochs shuffled passes of minibatches
minibatches, the next one each call, and collects again once it has served them all. Its count of
those served is its own, so a run whose update pools several calls (accumulation), or whose
optimizer holds or rejects an update, serves each collection whole, once a pass. Its carry, the
environments mid episode, stays on the device between collections. metrics reports the mean
return of the episodes that ended in the last collection.
ControlRollout.branch
Section titled “ControlRollout.branch”def branch(name: str) -> Iterator[None]A cooldown branch’s calls (Rollout’s branch), after which the rollout stands as the run left
it: its environments mid episode, the collection it serves and their order. Nothing it carries is
donated, so what the run left is kept as it is; a begun Simulator keeps its own environments
(dew.rl.simulator.branched).
ControlRollout.saved
Section titled “ControlRollout.saved”def saved() -> Saved | NoneWhat the rollout carries between calls, for the run’s checkpoint (Rollout’s saved): every
environment mid episode and its running return, the collection it serves its minibatches from,
their order, the collection’s mean ended return and how many of them it served; None before its
first call, and over a Simulator, whose environments live outside JAX and start afresh on a
resume.
ControlRollout.saved_shapes
Section titled “ControlRollout.saved_shapes”def saved_shapes() -> Mapping[str, object] | Nonesaved’s leaves’ shapes and dtypes, before any call; None over a Simulator.
ControlRollout.restored
Section titled “ControlRollout.restored”def restored(tree: Saved) -> NoneThe rollout carrying on from a saved tree; one saved before the rollout counted the minibatches
it served reads them off the updates at its next call, as the run that saved it did.
class Saved(TypedDict)What a ControlRollout carries between calls, as a checkpoint holds it (ControlRollout.saved):
every key but served, which one written before the rollout counted its own lacks.
episode_return
Section titled “episode_return”def episode_return( objective: ControlPPOObjective, env: Env | Simulator, params: Variables, key: jax.Array, steps: int, *, greedy: bool = True,) -> floatThe mean undiscounted return of the first episode of each of env’s environments, an Env or a
Simulator, over steps steps of the policy, its deterministic action where greedy: the whole
episode where steps is an episode’s length (dew.rl.evaluation.episodes).
tanh_normal_entropy
Section titled “tanh_normal_entropy”def tanh_normal_entropy( location: jax.Array, scale: jax.Array, key: jax.Array,) -> jax.ArrayThe squashed normal’s entropy as Brax estimates it: the normal’s, plus the tanh’s log-Jacobian at one draw, summed over the action.
tanh_normal_log_prob
Section titled “tanh_normal_log_prob”def tanh_normal_log_prob( raw: jax.Array, location: jax.Array, scale: jax.Array,) -> jax.ArrayThe log-density of the action tanh(raw), raw drawn from N(location, scale), summed over
the action: the normal’s log-density less log(1 - tanh(raw)^2), in its stable form
2 (log 2 - raw - softplus(-2 raw)).