Demem: Privacy-Enhanced Robust Adversarial Learning via De-Memorization

Balancing adversarial robustness and privacy in machine learning models is essential yet particularly challenging. Previous studies have shown that enhancing adversarial robustness through adversarial training often increases vulnerability to privacy attacks, revealing a fundamental tension between these two attributes. How to reduce privacy leakage in robust models has received little attention. Existing approaches like differential privacy (DP) remain ineffective in adversarial training, as it can severely degrade performance (e.g., test accuracy dropping from 50.87 % to 20.35 % on CIFAR-100). Through a detailed analysis, we find that such limitation arises because DP treats all training samples uniformly, failing to account for the diverse privacy risks and generalization abilities of individual samples. This oversight disproportionately impacts relatively low-risk samples which are often the typical samples that are essential for model robustness, leading to undesired performance degradation. To address this limitation, we propose DeMem, a novel method that selectively targets high-risk samples to achieve a better balance between privacy protection and adversarial robustness. By leveraging sample-wise granularity, DeMem minimizes privacy leakage without compromising robustness. Extensive evaluations across multiple datasets and adversarial training methods demonstrate that DeMem significantly reduces privacy leakage while maintaining robust performance against both natural and adversarial samples. (e.g., implementing DeMem with PGD-AT on CIFAR-10 reduces privacy leakage risk by 8 % without compromising both nature and robust accuracy). These results highlight DeMem's versatility, effectiveness, and broad applicability in enhancing the trustworthy attributes of machine learning models.