CaptainA self-study mobile app for practising speaking: task completion assessment and feedback with generative AI

We introduce the CaptainA mobile app, designed to meet the needs of second language (L2) learners engaged in self- study of Finnish, with potential applicability to other languages. Our app can provide automatic speaking assessment (ASA) of task completion in picture-based tasks, along with grading ex- planations and corrective feedback. It can also automatically generate pictures for visual tasks, providing users with unlim- ited practice opportunities. The mobile app is based on our framework that combines visual natural language generation (NLG), automatic speech recognition (ASR), and prompting large language model (LLM) for low-resource language. Our goal is to promote the development of next-generation speech- based computer-assisted language learning (CALL) systems ca- pable of providing automatic scoring with feedback for learn- ers, even when minimal speech data of L2 learners is available. While the mobile app demonstration is designed for Finnish, the app can also be tested in English.