Analysis of Various Self-Supervised Learning Models for Automatic Pronunciation Assessment

The advancement of Automatic Pronunciation Assessment (APA) systems has been significantly improved by Self-supervised Learning (SSL) models. However, despite these performance gains, there remains a lack of systematic research on effective utilization of SSL models and the explainability of their behavior in APA. This study aims to evaluate pronunciation with high accuracy using SSL models and to provide explanations for the scoring outcomes. To achieve this, we fine-tune various SSL models using multiple strategies, comparing their performance through extrinsic analysis to identify the key factors influencing performance improvements. Furthermore, intrinsic analysis is conducted using Principal Component Analysis (PCA) to gain insights into the model’s scoring patterns. Extrinsic analysis highlights the importance of strategic fine-tuning and acoustic similarity between fine-tuning and pre-training datasets. Intrinsic analysis reveals that different SSL models focus on distinct pronunciation features, with the Wav2Vec2.0 model capturing more advantageous information for APA. This study presents the first in-depth analysis of SSL models in APA, proposing a novel intrinsic analysis method based on feature distribution manifolds. We provide model-specific fine-tuning guidelines for APA tasks and recommend appropriate SSL models based on specific pronunciation assessment goals. This research significantly contributes to the future development of explainable APA systems based on SSL models.