Two-stage and Self-supervised Voice Conversion for Zero-Shot Dysarthric Speech Reconstruction

Dysarthria is a motor speech disorder commonly associated with conditions such as cerebral palsy, Parkinson’s disease, amyotrophic lateral sclerosis, and stroke. Individuals with dysarthria typically exhibit significant speech difficulties, including imprecise articulation, lack of fluency, slow speech rate, and decreased volume and clarity, which can hinder their ability to communicate effectively with others. We propose a two-stage Voice Conversion method to enhance the reconstruction of dysarthric speech. In the first stage, we develop a KNN-VC approach based on a same-gender-retrieval strategy to preliminarily repair the dysarthric speech. In this stage, we match the dysarthric speech only with normal speech of the same gender. In the second stage, we adapt so-vits-svc to restore the speaker’s timbre and improve the sound quality of the speech repaired in the first stage. Both objective and subjective evaluations were conducted on the dataset of the Low Resource Dysarthria Wake-Up Word Spotting Challenge (LRDWWS Challenge) shows that the proposed approach can achieve some improvements in terms of speaker similarity, speech intelligibility and naturalness for unknown speakers, and these evaluations also show our method has a good Zero-shot performance. Our audio samples can be accessed online <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>.