Teaching contact-rich tasks from visual demonstrations by constraint extraction

Contact-rich manipulation involves kinematic constraints on the manipulated object’s motion, typically with discrete transitions between constraints. Allowing the robot to detect and reason about these contact constraints can support robust and dynamic manipulation, but how can these contact models be efficiently learned? Purely visual observations are an attractive data source, allowing passive task demonstrations with unmodified objects. To use visual demonstrations for contact-rich tasks, we propose a method where object pose trajectories are clustered by contact modes, and then parameterized constraint types are selected and fit to minimize constraint violation. The fit constraints are then used to (i) detect contact online with force/torque measurements and (ii) change the robot policy with respect to the active constraint. We demonstrate the approach with real experiments, on cabling and rake tasks, showing the approach gives robust manipulation through contact transitions.