Dynamic Region Fusion Neural Radiance Fields for Audio-Driven Talking Head Generation

Since humans are sensitive to facial appearance and responsiveness, audio-driven talking head generation remains an ongoing challenge. The paper proposes DRMF-NeRF, a novel approach for generating audio-driven head generation by conditional Neural Radiance Fields (NeRF) audio, which can simultaneously achieve fast convergence of the model and generation of facial with natural motion. More specifically, we present an attention-based mechanism dynamic region fusion module that can effectively explore the potential connections and the degree of influence of audio on various face regions to generate a natural and accurate dynamic talking head. Furthermore, inspired by S3IM loss, we design an optimization function that combines the image pointwise loss and structural loss to enhance the quality of the generated image. Comprehensive experiments have shown that our method can generate realistic speech head videos, and its performance is superior to the baseline method.