Specialized Face Encoder for Audio-based Speech to Lip Generation

Videos that are directly translated and dubbed often fail to provide a natural audiovisual experience, making it crucial to develop methods that can accurately generate lip movements to match the audio. Self-supervised learning models like MAE have proven their robust transfer capabilities. This paper employs the pretraining method of MAE and proposes a new automatic lip-sync network. Additionally, we introduce a separate perceptual loss for the lip region of the generated faces to ensure accurate and clear lip movements. Our automatic lip-sync network consists of four components: video preprocessing, audio preprocessing, generator, and lip-sync discriminator, using facial images and audio features as inputs. We trained an MAE encoder and integrated it into the lip generator. The system was tested on two audiovisual datasets, and the results demonstrate that our proposed method achieves superior performance in both the visual quality of generated talking faces and their synchronization with the audio.