CrossModal-DEAF: a cross-modal adaptive fusion network for robust speech enhancement

This study addresses two key issues in speech enhancement: limited representation ability of single domain feature and mismatch between traditional loss functions and perceptual evaluation indicators. A new method is proposed, including two core innovations: (1) a cross-modal adaptive attention fusion mechanism based on a learnable sparse matrix to achieve dynamic mutual enhancement of time-frequency domain features; (2) a multi-objective perceptual perception optimization framework combined with non-guided indicators (such as STOI, PESQ), and reinforce learning through adaptive weighting strategies. Experimental results show that under low signal-to-noise ratio (-5–5 dB), the proposed method improved by 23.8% and 8.57% on STOI and PESQ indicators, respectively, which was significantly better than the existing method. This work provides an important reference for promoting the theoretical research and practical application of speech enhancement.