Cross-Modal Feature Enhancement Networks for Audio-Visual Event Localization

The task of audio-visual event (AVE) localization involves the temporal localization of both audible and visible events captured by camera sensors. However, the audio noise and visual background inherent in these sensors can obscure the distinction between event categories, or the inconsistency between audio and visual information may lead to incorrect predictions of event boundaries. To address these challenges, we propose a cross-modal feature enhancement network (CMFEN) designed to enhance modality-specific audio and visual features as well as overall AVE features. Specifically, our modality-specific feature enhancement approach employs cross-modal attentional mechanisms that allow the visual and audio modalities to focus on event-related features in the opposite modality. This approach also establishes intramodal dependencies in each modality, enhancing the temporal relevance of event-related audio and visual features. Additionally, our AVE feature enhancement scheme develops intermodal correspondences through audio-visual interaction fusion, aligning the feature representations of AVEs temporally. To further mitigate the interference of visual background noise, we have developed an audio-visual channel-spatial fusion (AVCSF) module that leverages audio-guided visual attention to improve the visual representation of the sounding target in the visual scene. Experiments conducted on the AVE dataset demonstrate that our method surpasses state-of-the-art methods in terms of AVE localization accuracy in both supervised and weakly supervised settings.