Improving Robot Social Perception in Human-Robot-Interaction Using Multi-modal Cues

This paper proposes a multi-modal framework for enhancing robot social perception in human-robot interaction applications. By integrating multiple sensory modalities-such as visual, auditory, and spatial data-robots can improve their understanding and responsiveness to human social cues. Our approach leverages advanced algorithms for object detection, face recognition, gaze tracking, and voice recognition, combined into a coherent system to improve interaction outcomes. A key result of this pipeline is the identification of the individual most engaged with the robot, enabling targeted interactions such as personalized dialogues and tailored assistive tasks. We demonstrate this framework's implementation in a socially assistive robot, showcasing notable enhancements in identifying and interacting with the most engaged user, thereby markedly improving both user engagement and the quality of interactions.