Enhancing Mispronunciation Detection with WavLM and Mixture-of-Experts Network

Mispronunciation Detection and Diagnosis (MDD) technologies are pivotal elements of Computer-Assisted Pronunciation Training (CAPT) systems. Despite the progress in deep learning that has enhanced the efficacy of pronunciation error detection, the capabilities and robustness of current systems must be further augmented, particularly when confronted with the scarcity of second language (L2) data and the multitude of error types. Additionally, the mother tongue background, age, gender, and pronunciation habits of L2 learners significantly influence the assessment process. Divergent from prior endeavors, this study leverages the Self-Supervised Pretraining (SSP) model WavLM to extract acoustic features and integrates a speaker classifier to distill L1-accent-related features. We introduce a Mixture of Experts (MoE) system that dynamically assigns weights and selects expert models in accordance with the extracted speaker information, enabling a more tailored adaptation to the phonetic traits of individual speakers. Experiments were conducted utilizing two publicly accessible datasets (TIMIT, L2-arctic), demonstrating consistent performance enhancement across various speaker profiles and bolstering the system's robustness. The optimal system achieved an F1 score of 78.48%, thereby substantiating the efficacy of our approach in MDD applications.