Towards multi-modal Learning by Demonstration: Combining depthless visual data with kinesthetic teaching

Human-human learning is usually conducted utilizing multiple modalities (verbal, visual, kinesthetic etc), which is proven to significantly improve the quality of communication between the teacher and the learner. Towards extending this idea to human-robot learning, we propose here a multi-modal scheme for robot Learning by Demonstration (LbD), that combines visual observation of the human hand from a depthless RGB camera, with enhanced direct kinesthetic teaching of the robot. The proposed method, exploits the visually perceived action of the human hand, to provide haptic guidance during kinesthetic teaching, facilitating corrections along the axes with the maximum uncertainty. The method is experimentally evaluated using an RGB camera and a UR10e robot, and compared against single-modal approaches, i.e. visually trained behavior and trained through kinesthetic teaching. Our method is shown to outperform the latter, in terms of both the quality of skill transfer, and the cognitive/physical load required for the training.