Motion capture and AR based programming by demonstration for industrial robots using handheld teaching device

In the industrial robots field, efficient and convenient programming methods have been a hot research topic. In recent years, immersive simulation technology has been developing rapidly in many fields, which provides new horizons for the development of industrial robots. This paper presents a HTC VIVE laser scan motion capture and Holohens augmented reality (AR) based interactive Programming by Demonstration (PbD) system for industrial robot. A portable Handheld Teaching Device (HTD) and its calibration algorithm are designed in the system. The portable HTD which is tracked by a laser motion capture system can be viewed as an AR robot end-effector to teach paths. Meanwhile, the AR robot can be simulated in real time during programing. In addition, the robot reproducing the operator's actions at the same position in space is the focus of programming. So, Multi-system registration methods are proposed to determine the relationship between robot systems, motion capture systems and virtual robot systems. Meanwhile, a path planning algorithm is proposed to convert the captured raw path points into robot-executable code. For unskilled operators, they can easily perform complex programming using the HTD. For skilled senior workers, their skills can be quickly learned by robots using the system.