Tri‐Plane Dynamic Neural Radiance Fields for High‐Fidelity Talking Portrait Synthesis

ABSTRACT Neural radiation field (NeRF) has been widely used in the field of talking portrait synthesis. However, the inadequate utilisation of audio information and spatial position leads to the inability to generate images with high audio‐lip consistency and realism. This paper proposes a novel tri‐plane dynamic neural radiation field (Tri‐NeRF) that employs an implicit radiation field to study the impacts of audio on facial movements. Specifically, Tri‐NeRF propose tri‐plane offset network (TPO‐Net) to offset spatial positions in three 2D planes guided by audio. This allows for sufficient learning of audio features from image features in a low dimensional state to generate more accurate lip movements. In order to better preserve facial texture details, we innovatively propose a new gated attention fusion module (GAF) to dynamically fuse features based on strong and weak correlation of cross‐modal features. Extensive experiments have demonstrated that Tri‐NeRF can generate talking portraits with audio‐lip consistency and realism.