Audio2Lip: Speech-Driven 3D Face Animation Based on Facial Motion Prediction and Face Reconstruction

High-fidelity 3D facial animation has a wide range of applications in virtual reality, movie and television production, and games. Existing speech-driven 3D facial animation methods usually require speech and 4D scanning data for training, but due to the scarcity of high-precision scanning data, it is difficult for these methods to capture a wide range of speech emotions and facial motion features from limited speech and scanning data, and thus it is difficult to generate rich and diverse 3D face animations. In addition, the generalization ability of these methods on Chinese and other languages is yet to be improved because the training data are mostly monolingual. To alleviate the scarcity of scanned data and improve the performance of face animation, we propose a new framework to learn speech and face motion representations on a large amount of audiovisual data, and then use speech coding as a control constraint to estimate motion features. The motion features are used to predict the portrait animation, which is then fed into a hierarchical representation network to reconstruct a 3D face sequence, and finally the 3D face sequence is smoothed with a smoothing postprocessing module to obtain a more reliable 3D face animation. The experiments show that our method performs better on the metrics of Identity Preservation, Mouth Synchronization, Image Quality, and Landmarks Distance of faces. Mean Opinion Score base on Motion Jittering proves that our method is able to generate temporally consistent animations. In addition, the ablation experiments also demonstrate the effectiveness of the joint training strategy and the smoothing post-processing module. This study has advantages in innovation and practicability, and provides new ideas and methods for speech-driven 3D face animation with low data volume, which is rich in research value and application prospects.