Audio-Visual Speech Synthesis Leveraging Capsule-Enhanced Generative Adversarial Network

Audio-visual speech synthesis (AVSS) is a rapidly gaining prominence in artificial intelligence (AI) due to its widespread applications. AVSS refers to generating the audio-visual speech of a target speaker from the audio input of a source speaker, while preserving the speech content. The sequential steps of the AVSS process include voice conversion (VC) for altering vocal characteristics from the source to the target speaker and audio-visual synthesis (AVS) to generate a video stream from the converted speech of the target speaker. However, with the progressive development of deep generative models, the domain of AVSS remains relatively unexplored. The existing literature predominantly focuses on autoencoder-based AVSS models, potentially limiting their ability to capture the diversity of the vocal features and also synthesize the realistic video output. This paper introduces a novel AVSS approach utilizing a generative adversarial network (GAN) model, enriched with a capsule network (Caps-Net) architecture to effectively capture relevant input speech features and produce more natural audio-visual output. The proposed framework is trained and tested on VoxCeleb2 and LRS3-TED datasets and the experimental outcomes exhibit the superiority of the proposed approach over the state-of-the-art (SOTA) models.