DiffAM-RGN: enhancing the transferability of adversarial makeup on facial recognition with diffusion model and learnable region

As facial recognition (FR) systems become increasingly prevalent across various domains, concerns regarding the unauthorized use and exploitation of facial data have intensified, underscoring the need for robust privacy protection mechanisms. While adversarial attack techniques have shown promise in countering such threats, existing methods often face limitations such as compromised visual quality, insufficient transferability, and poor performance in real-world blackbox scenarios. To address these issues, we present a novel adversarial makeup method, Enhancing the Transferability of Adversarial Makeup on Facial Recognition with Diffusion and Learnable Region (dubbed as DiffAM-RGN), capable of generating imperceptible, natural-looking, and highly transferable adversarial examples. Our approach utilizes a vision transformer-based region localization strategy to precisely apply makeup to targeted facial regions (e.g., eyes, lips, and skin). By integrating diffusion models with CLIP-driven style prompts, the proposed adversarial makeup method achieves fine-grained semantic control and ensures that the generated makeup aligns seamlessly with user-defined aesthetic goals. To further enhance imperceptibility and attack success rates, we introduce a combination of pixel-level and semantic alignment losses, allowing for refined adversarial makeup generation with high fidelity and natural visual quality. Additionally, an ensemble-based adversarial attack strategy improves transferability across black-box FR systems, including commercial platforms. Extensive experiments conducted on benchmark datasets demonstrate that our proposed DiffAM-RGN attack method outperforms state-of-the-art approaches, achieving superior visual quality, enhanced transferability, and a 6.37% improvement in average attack success rates compared to state-of-the-art methods. This work provides an innovative and practical solution for safeguarding facial privacy in an increasingly monitored world. The code will be available at https://github.com/Sreyna1/DiffAM-RGN.git.