Ivory: Adversarial Purification of Obfuscated Faces to Extract Soft-Biometrics using Diffusion Transformers

The proliferation of online face images has heightened privacy concerns, as adversaries can exploit facial features for nefarious purposes. While adversarial perturbations have been proposed to safeguard these images, their effectiveness remains questionable. This paper introduces IVORY, a novel adversarial purification method leveraging Diffusion Transformerbased Stable Diffusion 3 model to purify perturbed images and improve facial feature extraction. Evaluated across gender recognition, ethnicity recognition and age group classification tasks with CNNs like VGG16, SENet and MobileNetV3 and vision transformers like SwinFace, Ivory consistently restores classifier performance to near-clean levels in white-box settings, outperforming traditional defenses such as Adversarial Training, DiffPure and IMPRESS. For example, it improved gender recognition accuracy from $37.8 \%$ to $96 \%$ under the PGD attack for VGG16 and age group classification accuracy from $\mathbf{2. 1 \%}$ to $\mathbf{5 2. 4 \%}$ under AutoAttack for MobileNetV3. In black-box scenarios, IVORY achieves a $22.8 \%$ average accuracy gain. IVORY also reduces SSIM noise by over $50 \%$ at 1x resolution and up to $80 \%$ at $2 x$ resolution compared to DiffPure. Our analysis further reveals that adversarial perturbations alone do not fully protect against soft-biometric extraction, highlighting the need for comprehensive evaluation frameworks and robust defenses.