Beyond Replay: Automatic Corrections as Direct Policy Supervision for Robotic Manipulation

Automatic corrections offer direct policy supervision during assisted robotic reinforcement learning. We evaluate whether regressing on these actions improves learning beyond their use in replay. Uniform corrective distillation (UCD) adds actor regression on initial demonstrations and online corrections to an AutoSERL-derived soft actor–critic system. We compare four objectives on simulated spatula hanging and drawer opening with fixed and randomized starts and extend the randomized UCD–baseline comparison to five training seeds per method. Training stops when the shared process-success criterion is met or the budget is exhausted; each returned model receives 100 fresh test episodes with task assistance disabled. Across five seeds, UCD averaged 100.0% success on spatula and 91.2% on drawer, versus baseline means of 71.6% and 79.8%, with all five spatula UCD models achieving 100/100 successes. A separate comparison matched auxiliary batch size and update frequency; correction-inclusive supervision exceeded demonstration-only supervision in all six task–seed pairs. These results support direct actor regression for reusing automatic corrections under the evaluated stopping protocol.