A Novel Human-in-the-Loop Multimodal Intention Fusion Method for Human–Robot Interaction

Understanding human intention plays a crucial role in the research of human-robot interaction (HRI) for service robots. Although multimodal sensing methods have shown promising results in certain scenarios, the fusion strategy cannot flexibly adapt to user preference and dynamic environment. To address this limitation, this paper proposed a human-in-the-loop multimodal intention fusion (HIL-MIF) algorithm that introduces weight factors to assess the importance of each modality. By dynamically adjusting these weight factors based on user feedback, we achieved high accuracy in intention understanding, enhancing the personalization and reliability of the interaction. In addition, a multimodal service robot system was proposed that supports three interaction modalities including gaze, voice and gestures, which can be flexibly configured into different combinations. Validation experiment was performed on target object grasping tasks and collected user subjective evaluations. The experimental results demonstrate that the HIL-MIF proposed in this paper is superior to other commonly used multimodal fusion methods in terms of both accuracy and reliability. This paper paves the way for the future tri-co intelligent robot deployment in the service industry.