Collaborative Robot Teleoperation in Mixed Reality Environment for Inspection Tasks

Abstract Collaborative robots are increasing their presence in the industrial production context due to their flexibility in the tasks they can perform. However, programming and teleoperating them could be a difficult task from the perspective of an unqualified worker. In cases where the worker has to collaborate remotely with a robotic arm, the probability of collisions between the robotic arm and the environment increase. In this work, we describe the design and implementation of a mixed-reality multimodal interface for a worker to teleoperate a robotic arm by interacting with its holographic digital twin, while remaining aware of surrounding obstacles to avoid any collisions. To teleoperate the robot, we have developed a multimodal interface with which the user can communicate naturally with voice (through a semantics-based task-oriented dialogue system), and by performing manual actions on holographic objects. The application also provides audiovisual information to the user about the status of the collaborative robot as well as the actions commanded. We compute and command the behavior of the robotic arm with ROS (Robot Operating System) and “MoveIt!” (motion planning framework), and we represent computed trajectories and the real-time pose of the robot in the mixed reality environment.