Skip to content

dew.rl.aggregation

A student’s own steps labelled by a privileged teacher and aggregated into what the student learns from: DAgger (Ross, Gordon and Bagnell, arXiv 1011.0686), for any teacher policy and any student policy over one environment.

A student cloned from a teacher’s flights alone learns the teacher’s actions at the teacher’s states, and its own errors take it to states those flights never reached, where its errors grow. Here the student acts, so its data holds the states its own errors lead to, and at each of them the teacher, reading what the student cannot (a privileged view of the true state), answers the action it would take there: the step’s label (labelled). A rollout given a Teacher writes each label beside its step into the data it aggregates (dew.objectives.rl.dreamer.DreamerRollout’s replay, which also holds the teacher’s own flights it was given), and an objective imitates the label in place of the action taken (DreamerV3Objective(imitates=LABEL)), while what the student learns of the environment, a world model, reads the actions it took.

NameSummary
LABELThe field a labelled step’s teacher action is written as (labelled), which an objective imitates.
TeacherA privileged teacher: policy, a dew.rl.env.Policy over the step a student is given, reading its own views of it, and its variables.
labelledA policy over a student’s and a teacher’s variables (paired): student acting from the step and its memory, its extras, a dict, with teacher’s action at the same step beside them under LABEL.
pairedThe variables a labelled policy reads: the student’s and the teacher’s.

attribute source

LABEL = 'label'

The field a labelled step’s teacher action is written as (labelled), which an objective imitates.

dataclass source

class Teacher(policy: Policy, variables: Params)

A privileged teacher: policy, a dew.rl.env.Policy over the step a student is given, reading its own views of it, and its variables. It carries no memory: the view it reads holds the state.

function source

def labelled(student: Policy, teacher: Policy) -> Policy

A policy over a student’s and a teacher’s variables (paired): student acting from the step and its memory, its extras, a dict, with teacher’s action at the same step beside them under LABEL. The teacher draws from a key of its own, folded from the step’s.

function source

def paired(student: Params, teacher: Params) -> Params

The variables a labelled policy reads: the student’s and the teacher’s.