Toward Adaptive Spoken Dialogue Systems for Language Learning: Predicting Task Completion from Learning Process Data

Within the space of dialogue systems for language learning, the rapid advance of conversational artificial intelligence, powered among others by large language models, is currently driving innovations that enable learners of a second or foreign language (L2) to practice dialogic interaction at their own pace, including through functional tasks. Such language practice is particularly relevant when learners have limited opportunities to interact with more proficient speakers of that L2. However, to ensure a meaningful contribution of dialogue systems to L2 development, technology-mediated practice of dialogic interaction needs to be adapted to the needs and proficiency of learners. This requires accurate and transparent assessment of L2 performance that is both driven by theories about L2 acquisition and practically feasible with state-of-the-art technologies. The end goal is that task-based dialogue practice can be scaffolded through individualized feedback and other forms of learning support. This study models task performance in Language Hero, a game-based spoken dialogue system designed for Dutch-speaking learners who want to practice French as a L2. Using explanatory item response analysis, we explored to what extent the completion of functional spoken tasks can be predicted from learning process data, more specifically from fully-automated measures of L2 task performance. Data were drawn from 263 participants who completed a total of 739 tasks in the system, comprising 22,074 spoken responses. The results indicate that 12 fully-automated measures of previous task performance, including complexity, accuracy, fluency, and functional adequacy, as well as two measures of hint use, significantly predicted future task completion. A multilevel model with fixed and random effects accounted for 45% of the variance in task completion. This study demonstrates the potential of data-driven learner models for micro-adaptivity in dialogic technology-mediated practice while simultaneously highlighting the need to include complementary predictors as well as human evaluation.