SAM: A privacy-preserving framework for selective attribute masking in voice recordings

• Introduce Selective Attribute Masking (SAM) for voice privacy. • Use adversarial perturbations to mask age, gender, and accent. • Propose a composite loss for selective masking in multi-head models. • Achieve up to 74.5. • Balance privacy protection with speech recognition utility. Human voice is a rich source of information that can reveal a range of sensitive personal attributes, such as age, gender, and country of origin. With advances in Artificial Intelligence (AI), especially in speech processing, these personal attributes can now be inferred on a scale with high accuracy, raising serious privacy concerns. In fact, the ability to extract demographic or identification information from voice data poses risks related to surveillance, profiling, and misuse of personal data, highlighting the urgent need for privacy-preserving solutions in voice-based AI systems. Therefore, this paper proposes Selective Attribute Masking (SAM) , a new model-agnostic framework that uses gradient-based adversarial perturbations to suppress the inference of specific speaker attributes from voice recordings, while preserving the accuracy of non-target attributes and maintaining the utility of Automatic Speech Recognition (ASR). Experimental results using CommonVoice dataset demonstrate that SAM achieves selective masking success rates of up to 74.5 % for age, 59.4 % for gender, and 54.6 % for accent–substantially outperforming baseline methods. At the same time, voice utility (that is, ASR) remains largely unaffected, with the word error rate increasing by less than 3 % absolute under moderate perturbations. These findings demonstrate the effectiveness of our proposed framework (SAM) in balancing privacy and utility in voice-based systems.