General event detection algorithm of audio and video based on multimodal models

This paper focuses on the general event detection task of audio and video from the perspective of pre-trained feature extractor. To solve the problem that the existing work lacks the ability to effectively use multi-modal large model feature modeling in this downstream task, this paper proposes an audio and video general event detection algorithm based on AudioCLIP adapter and time-domain multi-scale fusion network based on cross-modal attention. The method consists of two training stages: 1) In the first stage, a linear adapter is used to adjust the features of the pre-trained model to adapt to the downstream task; 2) In the second stage, the multi-scale fusion network is proposed to strengthen the model's context-dependent modeling ability of events by using timing information of different scales. This method can effectively adapt to pre-trained large models and enhance the time series information fusion of audio and video features. The experimental results show that on the AVE dataset, the proposed method achieves 79.36% overall classification accuracy, which is 6.66% higher than that of the baseline method, and has strong event detection capability.