Joint Semantic and Spatial Feature Learning for Audio-Visual Speaker Tracking

Audio-visual speaker tracking, as a fundamental task in the field of multimodal perception, has broad application prospects in the domain of human-computer interaction. However, existing approaches primarily focus on spatial localization cues within audio-visual data, often utilizing only single-level features, which neglect the complementarity between semantic and spatial information. This creates challenges for current technologies in complex environments, such as occlusions, background interference, and environmental noise. To address this issue, this paper proposes a novel Audio-Visual Tracker based on Semantic-Spatial feature fusion (AVSS-Tracker). First, a semantic feature encoding method based on an audio-visual keyword spotting network is designed, capable of extracting semantic information related to the target from audio-visual signals. Then, a two-stage semantic-spatial feature fusion mechanism is introduced, which leverages a cross-attention mechanism to explore the correlations and complementarities of features across different levels and modalities, significantly enhancing feature representation capabilities and tracking accuracy. Extensive experiments demonstrate that AVSS-Tracker performs exceptionally well in complex scenes and under interference conditions, with the fusion of semantic and spatial features notably improving the performance of audio-visual speaker tracking.