dew.rl.aggregation
A student’s own steps labelled by a privileged teacher and aggregated into what the student learns from: DAgger (Ross, Gordon and Bagnell, arXiv 1011.0686), for any teacher policy and any student policy over one environment.
A student cloned from a teacher’s flights alone learns the teacher’s actions at the teacher’s states, and its
own errors take it to states those flights never reached, where its errors grow. Here the student acts, so its
data holds the states its own errors lead to, and at each of them the teacher, reading what the student cannot
(a privileged view of the true state), answers the action it would take there: the step’s label (labelled).
A rollout given a Teacher writes each label beside its step into the data it aggregates
(dew.objectives.rl.dreamer.DreamerRollout’s replay, which also holds the teacher’s own flights it was
given), and an objective imitates the label in place of the action taken
(DreamerV3Objective(imitates=LABEL)), while what the student learns of the environment, a world model,
reads the actions it took.
| Name | Summary |
|---|---|
LABEL | The field a labelled step’s teacher action is written as (labelled), which an objective imitates. |
Teacher | A privileged teacher: policy, a dew.rl.env.Policy over the step a student is given, reading its own views of it, and its variables. |
labelled | A policy over a student’s and a teacher’s variables (paired): student acting from the step and its memory, its extras, a dict, with teacher’s action at the same step beside them under LABEL. |
paired | The variables a labelled policy reads: the student’s and the teacher’s. |
LABEL = 'label'The field a labelled step’s teacher action is written as (labelled), which an objective imitates.
Teacher
Section titled “Teacher”class Teacher(policy: Policy, variables: Params)A privileged teacher: policy, a dew.rl.env.Policy over the step a student is given, reading its
own views of it, and its variables. It carries no memory: the view it reads holds the state.
labelled
Section titled “labelled”def labelled(student: Policy, teacher: Policy) -> PolicyA policy over a student’s and a teacher’s variables (paired): student acting from the step and
its memory, its extras, a dict, with teacher’s action at the same step beside them under LABEL.
The teacher draws from a key of its own, folded from the step’s.
paired
Section titled “paired”def paired(student: Params, teacher: Params) -> ParamsThe variables a labelled policy reads: the student’s and the teacher’s.