A Feasibility Study With In-the-Wild Data in Human Interaction Settings: Acoustic-Visual Fusion for Predictive Sound Source Positioning

Acoustic data inherently contains spatial information regarding sounds. Most robotic systems rely primarily on speech recognition results for control; however, the utilization of rich directional cues could enhance their functionality. Especially in human-robot interaction, both verbal content and spatial context in acoustic data are crucial. To capture such auditory context, we employ acoustic maps in this research. These maps represent soundscapes as 2D projections, allowing robots to integrate auditory spatial cues with visual perception. Using a custom acoustic-visual module, we collect egocentric data from four daily scenarios and generate corresponding acoustic maps. By demonstrating a predictive sound source positioning task, we confirm the potential of acoustic maps to capture dialogue-relevant information. Based on these findings, we anticipate potential applications for robots, such as identifying individuals attempting to engage with the robot in crowded environments or predicting turn-taking in multi-party conversations.