Audio-visual speech enhancement with multi-level feature deep fusion under low signal-to-noise ratio

To address the limitations in feature extraction and cross-modal fusion in audio-visual speech enhancement, a multistage deep fusion method was proposed for low signal-to-noise ratio (SNR) conditions. The method consisted of an audio-visual encoding network, a fusion network, and an auditory decoding network. A multi-branch collaborative unit (MCU) was introduced in the auditory encoder, along with an audio-visual attention fusion module (AVAFM) between each visual and auditory layer. A fusion weighting block (FWB) was also designed to optimize and dynamically weight features at each stage. Experiments on TMSV and LGRID datasets showed that the proposed method significantly improved PESQ and STOI scores under various low-SNR conditions. Compared to audio-only enhancement, average gains of 38.95% in PESQ and 33.92% in STOI were achieved at -5 dB, -2 dB, and 1 dB. These results demonstrate the method’s strong denoising ability and the effectiveness of visual information.