Towards Safe and Efficient Learning in the Wild: Guiding RL With Constrained Uncertainty-Aware Movement Primitives

Guided Reinforcement Learning (RL) presents an effective approach for robots to acquire skills efficiently, directly in real-world environments. Recent works suggest that incorporating hard constraints into RL can expedite the learning of manipulation tasks, enhance safety, and reduce the complexity of the reward function. In parallel, learning from demonstration (LfD) using movement primitives is a well-established method for initializing RL policies. In this paper, we propose a constrained, uncertainty-aware movement primitive representation that leverages both demonstrations and hard constraints to guide RL. By incorporating hard constraints, our approach aims to facilitate safer and sample-efficient learning, as the robot need not violate these constraints during the learning process. At the same time, demonstrations are employed to offer a baseline policy that supports exploration. Our method improves state-of-the-art techniques by introducing a projector that enables state-dependent noise derived from demonstrations while ensuring that the constraints are respected throughout training. Collectively, these elements contribute to safe and efficient learning alongside streamlined reward function design. We validate our framework through an insertion task involving a torque-controlled, 7-DoF robotic manipulator.