EMG-SENet: Multi-Modal Speech Enhancement With Electromyography Signals

Speech enhancement (SE) is crucial for improving the intelligibility and quality of noisy speech in applications such as speech recognition and assistive technologies. While traditional SE methods often fail in complex noise environments, deep learning-based approaches and multi-modal techniques have shown significant progress. However, existing multi-modal SE methods, such as audio-visual approaches, are limited in scenarios where visual data cannot be captured. EMG-SENet-basic explores Electromyography (EMG) signals as a robust complementary modality for SE, leveraging their resistance to environmental noise and potential for non-invasive acquisition. In addition, we propose an improved multi-modal SE framework named EMG-SENet-improved featuring a frame-level cross-attention mechanism for dynamic feature fusion, an adaptive weighting mechanism to better integrate amplitude information from both modalities, and the Con-TF-Mamba architecture to enhance local feature extraction in time and frequency domains. Furthermore, Temporal Masking, used to simulate frame loss, improves robustness under real-world conditions. Experimental results demonstrate the proposed method’s superior performance in enhancing SE quality and intelligibility, addressing key challenges in practical applications with degraded or incomplete data.