Evaluating the Accuracy of a Developed Machine Learning-Based Arabic Pronunciation Model for Beginner Level

The use of artificial intelligence (AI) in language learning has expanded rapidly, offering new methods to support speaking skill development. In Brunei, Arabic is widely taught as a second language, but learners often struggle with accurate pronunciation due to limited exposure to native speech environments. Recent advances in speech recognition and machine learning (ML) provide opportunities to bridge this gap, particularly through the use of lightweight, game-integrated pronunciation models. This study evaluates the accuracy performance of a machine learning-based Arabic pronunciation model developed using Google’s Teachable Machine and implemented within a Unity-based educational game. Drawing on prior work that emphasizes the importance of evaluating speech models with real-world learner input, this study focuses specifically on pronunciation accuracy among beginner-level Arabic learners. The system was tested with 35 non-native Arabic learners in Brunei, each of whom completed a 20-word pronunciation task. Accuracy score (%) and average attempts per word were calculated. Descriptive statistics revealed a mean accuracy of 34.86% (SD = 14.22), and a mean attempt count of 2.17 (SD = 0.57). A boxplot highlighted variation in learner performance, while Pearson correlation analysis showed a weak, non-significant negative relationship between accuracy and number of attempts (r = –0.215, p = 0.214). Findings suggest that while the model demonstrates functional classification capability, further refinement is needed to improve reliability. This research contributes to the growing field of AI-assisted language learning by offering a foundation for developing accurate, accessible pronunciation feedback tools for Arabic learners.