Audio-driven Talking-face Synthesis based on 3D Gaussian

Audio-driven talking-face synthesis is of significant importance in various application scenarios, such as remote meetings, AR/VR, and digital humans. Currently, the work in this field can be broadly divided into two categories: implicit representation-based methods and explicit representation-based methods. Implicit representation-based methods often use neural network models to represent human face. For example, NeRF (Neural Radiance Field) uses MLP (Multi-Layer Perceptron) to represent human face, and by inputting the camera pose, NeRF renders a face image from that viewpoint. When training with audio features, it can render face images with different facial movements. However, due to the lack of editability and slow rendering speed of implicit representations, the industry has been exploring new representation methods. Explicit representation-based methods commonly use 3D Gaussian representations of the face, which offers high editability and nearly real-time rendering speed, making these methods surpass implicit-based approaches in many metrics. However, it is difficult to incorporate other signals during the rendering optimization process. To address this issue, this paper combines 3D Gaussian representations with the 3D head model FLAME (Faces Learned with an Articulated Model and Expressions). The FLAME model represents the face's shape and facial movements with a set of parameters. In this paper, audio signals are first transformed into parameters representing facial movements, which are used to initialize the 3D model generated by the parameters, followed by rendering optimization. This paper proposes a 3D Gaussian-based audio-driven lip-sync video generation system. It consists of two main modules: the audio-to-facial movement module and the 3D Gaussian rendering module. The audio-to-facial movement module uses the extracted audio features to generate parameters that control the FLAME model, converting the audio input into natural and reasonable lip shapes and facial movements. The 3D Gaussian rendering module initializes the 3D Gaussian with the FLAME model, optimizes the rendered image from coarse to fine, and finally stitches the generated images together to create a continuous video. Experiments show that this system can generate lip-synchronized, accurate, high-fidelity videos.