Aperture: a patch-aware framework for joint forgery detection and localization

Abstract The proliferation of advanced generative models presents a significant threat to the authenticity of digital visual content, creating an urgent demand for robust forgery detection technologies. A primary challenge lies in developing a unified and generalizable framework that can not only classify an image as authentic or forged but also precisely localize the manipulated regions, all while maintaining computational efficiency and adapting to unseen manipulation techniques. To address these challenges, we introduce Aperture, a lightweight and effective framework for joint deepfake detection and localization. Our framework integrates a novel patch-aware classifier that fuses local and global features to enhance classification robustness. For localization, a segmentation module guided by conditional queries is employed to generate fine-grained masks of tampered regions. Crucially, we propose a pseudo-label-guided test-time adaptation mechanism, which leverages the classifier’s patch-level predictions to dynamically supervise the segmentation head. This allows the model to improve its robustness against out-of-distribution data encountered during testing.