Unsupervised End-to-End Accented Speech Recognition Under Low-Resource Conditions

Accents, characterized by deviations from standard pronunciation, often lead to a sharp decline in the performance of speech recognition systems. This issue becomes even more serious when dealing with unsupervised accents under low-resource conditions. And the effective utilization of limited unsupervised accent speech data to enhance accent-robust ASR remains largely unexplored and presents significant challenges. In this study, we introduce a novel approach termed Joint Unsupervised and Supervised Data Training (JUSDT) to handle this issue. In JUSDT, the supervised ASR training and unsupervised representation learning are treated as two distinct tasks but jointly trained in a single training process. Building upon JUSDT, we further explore two additional variants: Phoneme-Quantizer JUSDT and JUSDT+, which respectively employ phoneme codebook learning to generate accent-invariant representations for accent normalization, and enable supervised training for unsupervised accent speech data to well-incorporate both textual and acoustic contextual information. Our experiments are performed on both the Librispeech dataset and accented English ASR tasks. Results demonstrate that the proposed methods outperform our strong baseline by relative 3.2% to 9.3% word error rate reductions across multiple test sets