Advanced defense against GAN-based facial manipulation: A multi-domain and multi-dimensional feature fusion approach

Powerful facial image manipulation offered by encoder-based GAN inversion techniques raises concerns about potential misuse in identity fraud and misinformation . This study introduces the Multi-Domain and Multi-Dimensional Feature Fusion (MDFusion) method, a novel approach that counters encoder-based GAN inversion by generating adversarial samples. Firstly, MDFusion transforms the luminance channel of the target image into spatial, frequency, and spatial-frequency hybrid domains. Secondly, we use the specifically adapted Feature Pyramid Network (FPN) to extract and fuse high-dimensional and low-dimensional features that enhance the robustness of adversarial noise generation. Then, we embed adversarial noise into the spatial-frequency hybrid domain to produce effective adversarial samples. Finally, the adversarial samples are guided by our designed hybrid training loss to achieve a balance between imperceptibility and effectiveness. Tests were conducted on five encoder-based GAN inversion models using ASR, LPIPS , and FID metrics. These tests demonstrated the superiority of MDFusion over 13 baseline methods , highlighting its robust defense and generalization abilities . The implementation code is available at https://github.com/LuckAlex/MDFusion . • A novel framework to define and evaluate adversarial examples for encoder-based GAN inversion. • Multi-domain and Dimensional Features fusion (MDFusion) method to generate adversarial noise. • Extensive assessments showing MDFusion preserves image quality while thwarting attacks.