Efficient speech denoising using CleanUNet optimized with Mamba and hybrid spectral loss

Abstract Speech enhancement aims to recover clean speech from signals degraded by noise and adverse acoustic conditions, such as background interference and echo. CleanUNet has emerged as an effective solution for causal speech denoising, but its high computational demands limit deployment on resource-constrained devices. In this work, we propose an optimized version of CleanUNet that significantly reduces model complexity while maintaining competitive performance. We replace the original Transformer bottleneck with a Mamba module, which offers efficient sequence modeling with linear time complexity. Additional architectural improvements include batch normalization, lightweight channel attention, alternative activation functions, and a novel hybrid loss function that combines Short-Time Fourier Transform and Wavelet-based components. Our best models achieved up to 86% parameter reduction relative to the original CleanUNet baseline with minimal performance degradation, and an ultralight version reduced parameters by up to 98% while still delivering competitive results in terms of Perceptual Evaluation of Speech Quality (PESQ) and Short-Time Objective Intelligibility (STOI). A comprehensive Pareto analysis illustrates the trade-offs between model size and speech quality, providing a range of configurations suited for different application scenarios, including real-time and embedded environments.