DMAFN: Dynamic Multimodal Attention Fusion Network for Audio-Visual Event Localization

Audio-Visual Event Localization (AVE) seeks to pinpoint the time periods during which particular events take place by simultaneously examining audio and visual modalities, requiring effective fusion of multimodal information. However, existing methods often struggle with effective fusion of multimodal information, resulting in suboptimal performance. To overcome these limitations, we introduce the Dynamic Multimodal Attention Fusion Network (DMAFN), featuring a novel Dynamic Multimodal Attention Fusion module. This module consists of four essential blocks: two Mixture-of-Head attention blocks, a Dynamic Cross Modal Attention block, and a Contextual Cross Modal Attention block. Specially, it introduces a learnable dynamic weighting mechanism that adaptively allocates attention across multiple heads based on multimodal contextual information, enhancing the model’s capacity to capture both modality-specific and cross-modal interactions, and improving event localization. Extensive experiments on the AVE benchmark dataset demonstrate that DMAFN surpasses current state-of-the-art methods. Furthermore, ablation studies confirm the dynamic gating mechanism’s effectiveness in enhancing model adaptability, robustness, and generalization capabilities. Our research highlights the considerable potential of Dynamic Multimodal Attention Fusion mechanisms in advancing audio-visual event localization.