Video Falsity Detection Based on Multimodal Feature Fusion

Watching short videos has become one of people's daily habits. Therefore, identifying false videos is very important and necessary. Most of existing work has been focusing using visual or/and audio features. In this paper, we propose a false video detection model by deeply integrating multimodal features, FAMFVD for short, including text, audio, and visual. During the feature extraction, the model fully explores the implicit information in each modality, with the audio modality considering not only the audio spectral features but also extracting the corresponding speech text. The visual information includes keyframe features, video segment features, and video text content. Then, a multi-granularity feature fusion module is introduced, which not only considers the coarse-grained semantic feature fusion between keyframes and their corresponding audio text and embedded text but also integrates fine-grained features that include details within the modalities. Subsequently, the model dynamically identifies the correlation between different modality features through an attention mechanism, adaptively optimizing the information weight of different modalities. Experimental results show that the model proposed in this work has significant advantages in terms of accuracy, precision and F1 score, and especially, achieves better recall rate in comparison with SOTA.