Combining Deterministic Enhanced Conditions With Dual-Streaming Encoding for Diffusion-Based Speech Enhancement

Score-based diffusion models for speech enhancement (SE) need to incorporate correct prior knowledge as reliable conditions to generate accurate predictions. However, providing reliable conditions using noisy features is challenging. One solution is to use features enhanced by deterministic methods as conditions. However, the information distortion and loss caused by deterministic methods might affect the diffusion process. In this paper, we first investigate the effects of using different deterministic SE models as conditions for diffusion. We validate two conditions depending on whether the noisy feature was used as part of the condition: one using only the deterministic feature (deterministic-only), and the other using both deterministic and noisy features (deterministic-noisy). Preliminary investigation found that using deterministic enhanced conditions improves DNSMOS and UTMOS on real data, while the choice between using deterministic-only or deterministic-noisy conditions depends on the deterministic models. Based on these findings, we propose the deterministic-diffusion unified model for SE to more effectively utilize both conditions. Moreover, we found that fine-grained deterministic models have greater potential in objective evaluation metrics, while UNet-based deterministic models provide more stable diffusion performance. Therefore, we also propose a deterministic model that combines coarse- and fine-grained processing for the diffusion. Experimental results on CHiME4 show that the proposed models effectively leverage deterministic models to achieve better SE evaluation scores on the DNSMOS and UTMOS for real evaluation sets. In addition, the proposed deterministic model proves to be more stable than other deterministic models when it is used for diffusion.