Phonological level wav2vec2-based Mispronunciation Detection and Diagnosis method

The automatic identification and analysis of pronunciation errors, known as Mispronunciation Detection and Diagnosis (MDD) plays a crucial role in Computer Aided Pronunciation Learning (CAPL) tools such as Second-Language (L2) learning or speech therapy applications. Existing MDD methods relying on analysing phonemes can only detect categorical errors of phonemes that have an adequate amount of training data to be modelled. Due to the unpredictable nature of pronunciation errors made by non-native or disordered speakers and the scarcity of training datasets, it is unfeasible to model all types of mispronunciations. Moreover, phoneme-level MDD approaches can provide only limited diagnostic information about the error made. To address this, in this paper, we propose a low-level MDD approach based on the detection of phonological features. Phonological features break down phoneme production into elementary components that are directly related to the articulatory system leading to more formative feedback for the learner. We further propose a multi-label variant of the Connectionist Temporal Classification (CTC) approach to jointly model the non-mutually exclusive phonological features using a single model. The pre-trained wav2vec2 model was employed as a core model for the phonological feature detector. The proposed method was applied to L2 speech corpora collected from English learners from different native languages. The proposed phonological level MDD method was further compared to the traditional phoneme-level MDD and achieved a significantly lower False Acceptance Rate (FAR), False Rejection Rate (FRR), and Diagnostic Error Rate (DER) over all phonological features compared to the phoneme-level equivalent. • We proposed a novel Speech Attribute detection method that utilizes the wav2vec2 speech representation pre-trained model which outperforms the current state-of-the-art methods. • We proposed a novel variant of the Connectionist Temporal Classification (CTC) loss function, specifically designed for multi-label scenarios, to handle the non-mutually exclusive nature of speech attributes. • We proposed a low-level approach for Mispronunciation Detection and Diagnosis (MDD) that leverages the speech attribute detection method. • The proposed MDD approach outperforms state-of-the-art phoneme-level MDD approaches. • The proposed speech attribute-based MDD provides a detailed description of pronunciation errors, directly related to the articulatory system, enabling the construction of formative feedback.