SAM-Wav2lip++: Enhancing Behavioral Realism in Synthetic Agents Through Audio-Driven Speech and Action Refinement

Digital human generation is a forward-looking field in technology. Despite significant progress in the generation of speaking facial videos, many challenges remain unaddressed. Issues such as unnatural head movements, distorted expressions, artifacts in generated videos, and uncoordinated limb movements persist. Most current efforts are focused on specific individuals, with enhancements often limited to head movements without further advancing the overall behavioral actions of digital humans. In this context, we introduce a new dataset, CFMD, and a novel model, SAM-Wav2lip++, capable of generating consistent, audio-synchronized lip and behavior action videos from a single reference image of any identity. This work features three main innovative components: (1) a contrastive lip-sync discriminator for precise lip synchronization, (2) a generator for the synthesis of sound-action consistency, and (3) the SAM module for facial refinement operations. Through extensive experiments and user studies, our results demonstrate that our model can synthesize digital human videos of impressively high perceptual quality that accurately sync lip movements and behavioral actions with the input audio, substantially outperforming the state-of-the-art baselines evaluations.