Boosting Audio-Visual Speech Enhancement with Lip-Movement Speech Synthesis

In this study, we revisit the challenge of speech enhancement in environments characterized by low signal-to-noise ratios (SNRs) and complex noise conditions. Conventional methods primarily depend on audio signals alone, yet their effectiveness diminishes under these conditions. Recent research has incorporated lip movements as additional cues to enhance speech quality, but these approaches often lack a clean training target. To address this limitation, we introduce a novel framework that integrates speech enhancement with lip-movement-based speech synthesis. By employing speech directly synthesized from lip movements as a supplementary task to guide the learning process, our approach notably improves speech intelligibility and quality in challenging low SNR environments. In comparison to traditional audio-visual speech enhancement models, our method exhibits significant performance improvement.