Multimodal Framework for Deep Fake Detection and Content Moderation Using CNN, ViT, and Audio-Visual Analysis

This paper presents a multimodal framework for deepfake detection and content moderation, integrating audio, video, and text analysis to improve detection accuracy and robustness. The framework consists of seven key components: audio extraction, deepfake detection using CNN, audiovisual deepfake detection, metadata and caption extraction, advanced feature extraction using CNN and ViT, text-to-speech transcription in Bengali, and hate speech detection from images. Audio and video features extracted using CNN and ViT are combined to compute a probabilistic score for deepfake detection, while OCR and NLP models analyze transcribed text for hate speech. The system is designed for real-time multimedia analysis, making it suitable for social media content moderation, forensic investigations, and misinformation detection. Experimental evaluations on benchmark datasets such as Celeb-Df, and DFDC demonstrate high detection accuracy, outperforming traditional single-modal approaches. The framework is optimized for scalability and efficiency, enabling large-scale multimedia analysis with minimal latency. By integrating advanced video processing, speech analysis, and text recognition, this work contributes to multimodal AI research, offering a comprehensive and scalable solution for detecting manipulated media and ensuring digital content integrity in real-world applications.