Explainable Manipulated Videos Detection Using Multimodal Large Language Models

Misinformation detection is receiving significant attention due to their impact on modern social stability. Extensive research has been conducted on multimodal misinformation detection, employing either deep learning approaches or large language models (LLMs). However, existing literature on explainable LLM-based methods remained unexplored when applied to video data. Furthermore, most of the existing datasets focus on facial manipulation and human speech, whereas social media content encompasses a wide range of manipulations such as altering background context and color, adding or removing entities, editing audio and text. This paper proposed a framework based on multimodal large language models (MLLMs) for explainable manipulated videos detection. The framework involves fine-tuning two MLLMs for rationales generation and making final predictions, respectively. In addition, pre-trained encoders, cross-attention mechanisms, and a fusion gate are utilized to extract deeper features for fine-tuning the second MLLM. The goal is to predict manipulated content, identify the specific attack applied to the input video, and provide detailed explanations. This research discusses preliminary findings in literature, presents initial experimental results, and outlines key directions for future work.