AT2D: An Automated Testing Framework for Deepfake Detection Models

Deepfake videos are rapidly spreading across the internet, posing significant threats to public interests. To counter this issue, researchers have developed various deep learning-based deepfake detection methods, which have demonstrated excellent performance. However, these detection methods are typically evaluated based on relevant benchmarks that often fail to fully reflect the complexity of real-world scenarios, particularly when it comes to systematically measuring the various video distortions caused by uncontrollable factors in everyday life. To address this problem, this paper proposes an automated testing framework that constructs three types of metamorphic relations-focusing on background color, lens distortion, and weather conditions-to simulate various video distortions and assess their impact on detection models. To validate the effectiveness of this framework, we conducted experiments using two popular deepfake detection methods. Statistical results indicate that under our testing framework, the detected error rates ranged from 21% to 74%.