Mandarin speech reconstruction from surface electromyography based on generative adversarial networks

The loss of speech function due to conditions such as laryngectomy and vocal cord paralysis significantly impacts the quality of life for patients. Achieving effective communication for these patients is a goal pursued by researchers. This study primarily explores a method for reconstructing Mandarin speech based on voice-related neck and facial surface electromyography (sEMG). Neck and facial sEMG signals and speech waveform were synchronously collected during normal speech production. A speech reconstruction model for Mandarin speech, based on Generative Adversarial Networks (GANs) and multi-scale feature extraction from sEMG, was developed. Both subjective and objective evaluations were conducted to assess the speech reconstruction performance of the model. The evaluation results indicate that the model effectively reconstructs speech from neck and facial sEMG signals. The reconstructed speech closely matches the original in terms of spectrogram and fundamental frequency, with mel-cepstrum distortion of 8.45 dB, log F0 RMSE of 0.40, F0 correlation coefficient of 0.71 and F0 voiced/unvoiced estimation accuracy of 0.80. The character error rate of the reconstructed speech is 0.32, while the tone error rate is 0.26. The subjective listening test results show that the naturalness of the reconstructed speech is acceptable, with a mean opinion score greater than 3. This study demonstrates the potential of deep learning techniques in effectively reconstructing Mandarin speech from sEMG. • A GAN-based method directly reconstructs Mandarin speech from sEMG signals. • The approach effectively captures tonal and pitch variations in Mandarin speech. • Experimental results show smooth Mandarin speech reconstruction from sEMG. • The method offers potential for voice rehabilitation in patients with voice loss.