A Multimodal Human-Robot Interaction Control System

With the rapid advancement of intelligent manufacturing and robots operating in hazardous environments, collaborative robots are increasingly replacing humans in critical tasks. However, the operational efficiency of robots still lags behind that of humans, primarily due to suboptimal human-robot interaction (HRI). In this paper, we present a multimodal HRI control system comprising a multimodal interaction control box and corresponding interaction methods, which can effectively improve the efficiency of multimodal interaction during remote robot operation. First, we design a highly integrated and portable HRI control box, facilitating the efficient deployment of multimodal interaction devices anytime and anywhere. Second, we develop a continuous control scheme based on hand gestures and eye gaze that enables the operator to precisely control the robot during teleoperation to accomplish tasks. Finally, we propose an adaptive multimodal fusion method that dynamically adjusts fusion weights based on the operator's habits, thereby accommodating various users and enhancing the system's robustness. In the experiment, a cooperative four-rotor robot with high flexibility and freedom is selected as the control object. The results show that the system has the characteristics of easy deployment, comfortable operation, low learning cost, high efficiency, and good adaptability to different operators.