Generalizable Spatiotemporal Deepfake Detection with Late-Fusion

Recently, deepfakes have been spreading at an alarming rate, raising concerns about trust and privacy in social media, where misinformation is easily spread. Deepfakes can be used for various malicious reasons, such as for manipulating criminal evidence, blackmailing, or even identity theft. To combat this, many deepfake detection models have been developed. However, these models often struggle with poor generalizability across different datasets and deepfake types. In this paper, a spatiotemporal model is proposed to enhance generalization, using a late-fusion scheme where a spatial and a temporal module separately analyze spatial and temporal features, such as inconsistent lighting or color discrepancy. The spatial module is a Multi-Scale Vision Transformer (MViTv2), which focuses on spatial anomalies within frames. The temporal module is a Temporal Transformer which focuses on inconsistencies between frames in a video. The datasets used are FaceForensics++ (FF++) and CelebDF. The experiments include intra-dataset evaluations, where the model is trained and tested on one dataset, and cross-dataset evaluations, where it is trained on one dataset and tested on another, enabling us to measure the model’s generalizability. The model outperforms state-of-the-art models, achieving 91.0% AUC when trained on FF++c23 and tested on CelebDF.