Voice2Visage: Deciphering Faces From Voices

The human voice carries valuable cues about an individual’s identity and emotions. A more intriguing question emerges: can one’s facial appearance be deduced from their voice alone? Existing efforts have primarily focused on exploring the relationship between natural audio and visual data, with limited attention given to the specific biometric domain of speaker-voice and face correlation. This study seeks to model the facial-related information embedded within the voice and ultimately predict an unknown person’s appearance solely based on their unheard voice. This task presents several challenges: firstly, while natural sounds exhibit significant variability, human voices often share similar frequencies, complicating the establishment of mappings between them. Secondly, generating faces from a voice presents an ill-posed problem, as details such as makeup and heap pose cannot be inferred from voice alone. In this paper, we introduce a novel framework named Voice2Visage, designed to tackle this task by leveraging self-supervised cross-modal and intra-modal learning to predict faces corresponding to input voice. To ensure the feasibility of our method, we optimize existing algorithms in automated dataset collection. Additionally, we systematically design experiments to test the usability and stability of commonly used quantitative metrics in the field of facial identity comparison. The results validate the close semantic association between the generated face and the reference one, showcasing its reliability. Our work provides a fresh perspective on exploring the depth of physiological characteristics concealed within human voices and the intricate interplay between appearance and voice. Our code is available at https://github.com/colaudiolab/Voice2Visage.