NeRF-RA: Neural Radiance Field with Robust Data Processing and Attention-Based Talking Head Synthesis

This paper presents NeRF-RA, a novel Neural Radiance Fields (NeRF) based framework for generating high fidelity Chinese lip-sync talking head video. Existing NeRF-based methods for talking head synthesis impose strict requirements on input data: the audio must be noise-free, and the person in the image should be clearly distinguishable from the background. Failure to meet these conditions can lead to convergence issues and artifacts in the generated video. We introduce a vocal separation module to extract clean vocals audio and a cascaded inpainting module to isolate a natural background image. To achieve accurate lip sync, we employ an attention-based face-audio encoder to capture relevant features between audio and face. This module is trained on a large-scale Chinese dataset, which enhances its effectiveness in extracting Chinese audio features. Moreover, Stochastic Structural Similarity (S3IM) loss is integrated into the training process. Unlike traditional pixel-wise losses (e.g., MSE), S3IM leverages stochastically sampled pixel patches to capture spatial coherence and structural patterns. This approach enables the generation of visually consistent and synchronized lip movements. Extensive experiments demonstrate that NeRF-RA achieves competitive performance in generating realistic talking face videos compared to state-of-the-art methods, including realism and synchronization.