AVF-Lip: High-Fidelity Talking Face Generation via Audio-Visual Fusion

Talking face generation aims to create realistic facial videos with lips precisely synchronized to the input audio, finding broad applications in human-computer interaction, virtual avatars, and video conferencing. Despite significant advancements in previous works, existing methods continue to face challenges in achieving accurate lip synchronization and high visual quality. To address these challenges, we propose a talking face generation framework named AVF-Lip that supports higher resolution and accurate lip synchronization. Specifically, an audio-visual fusion module (AVFM) is first customized to improve lip synchronization performance, and this is further enhanced by introducing a dynamic and fine-grained face mask. Subsequently, to address the problem of lip distortion in high-resolution facial images, we design a set of loss functions based on face sketches to enhance the generation of the lip areas. Extensive experiments demonstrate that our AVF-Lip outperforms state-of-the-art methods in lip synchronization and delivers competitive visual quality, positioning it as a promising solution for high-resolution and high-fidelity talking face generation. Project page: https://wmd-1.github.io/