Tackling the challenge of construct validity in assessing newly arrived learners’ receptive skills: investigations on a multilingual online-based diagnostic tool

Introduction This study presents the theoretical rationale and first empirical findings on Allrad-M, a newly developed multilingual procedure for the assessment of receptive linguistic skills. The tool is designed to enhance test fairness and construct validity when evaluating newly arrived learners’ listening and reading comprehension. Unlike monolingual diagnostic instruments, Allrad-M enables learners to switch flexibly between German, Ukrainian, Russian, and English, allowing for a more accurate assessment of comprehension skills regardless of their proficiency in German as a second language. Methods The exploratory study is based on two data sources. First, ten screen recordings of learners’ interactions with Allrad-Mwere analyzed to examine how participants used their linguistic repertoires when processing texts and responding to test items. Second, a semi-structured interview was conducted with a teacher who implemented the tool in classroom practice, providing professional insights into its diagnostic potential. Results The analysis of the screen recordings shows that learners actively mobilized their multilingual resources while working with the tool. Language choices were shaped by context (reception vs. assessment) and modality (reading vs. listening). Case analyses further highlight individual strategies in the use of multiple languages. The teacher interview indicates that Allrad-M reveals aspects of learners’ potential that often remain undetected in monolingual assessments. Discussion The findings suggest that Allrad-M can strengthen formative assessment practices for newly arrived learners by recognizing multilingual repertoires as resources rather than obstacles. Future development should focus on integrating additional languages and providing targeted teacher training to support the implementation of multilingual diagnostic tools.