DHAT: Dynamic Head Allocation Transformer for Audio-Visual Event Localization

Audio-Visual Event Localization (AVE) aims to identify temporal segments where specific events occur by jointly analyzing audio and visual modalities. Traditional approaches often rely on fixed attention mechanisms or simple feature fusion strategies, which fail to adaptively capture the intricate relationships between modalities. In this paper, we propose DHAT (Dynamic Head Allocation Transformer), a novel Transformerbased framework designed to dynamically allocate attention resources across modalities and temporal dimensions. The core innovation of DHAT lies in its Dynamic Head Allocation (DHA) mechanism, which adjusts the contribution of each attention head based on input features. This enables the model to allocate computational focus adaptively, enhancing its ability to model modality-specific and cross-modal interactions. Additionally, DHAT incorporates a cross-modal attention module that leverages dynamically weighted features to facilitate finegrained information exchange between audio and visual streams. Extensive experiments on the AVE benchmark dataset demonstrate that DHAT outperforms state-of-the-art methods, achieving superior event localization accuracy. Ablation studies further validate the effectiveness of the DHA mechanism in improving model adaptability and robustness. Our work highlights the potential of dynamic attention mechanisms for advancing multimodal learning tasks.