Wav2Lip Bridges Communication Gap: Automating Lip Sync and Language Translation for Indian Languages

Translating spoken speech in videos from one language to another is known as audio-visual translation (AVT). This paper describes the implementation of an automated AVT and lip-synced dubbing application. It addresses the difficulty of synchronizing mouth movements with translated speech by building a web application that synthesizes the speaker’s lip movements to match translated audio. Using ASR models, the speech from the source video is converted to text, translated into several languages, and then automatically synthesized into speech in the target language. A lip synchronization model, Wav2Lip, is used to alter the mouth movements in the video to correspond to the target language. We compare our work with two wellknown ASR systems: Wav2vec 2.0 and Google Speech Recognition. Wav2vec 2.0 performs better with the lesser average WER% of 15.38 and is used in our final web application. The performance of the video dubbing component is discussed with the generated speech in Tamil, Telugu, Hindi, and English, and we determine that our generated videos outperform the existing ones. Our proposed AVT application is user-friendly for a wide variety of speakers, utilizing readily available TTS systems instead of training on an individual speaker’s voice.