The relationship between AI-chatbots use, student assessment performance and learning outcomes in higher education

Abstract In this study in German higher education, we draw on the first panel of dataset from a longitudinal assessment of domain-specific (DOM) Critical Online Reasoning (COR) in economics in the winter terms 2023/24 (t0) and 2024/25 (t1). DOM-COR tasks were completed as a digital assessment, combining open web search, information evaluation, and reasoning to solve domain-specific scenarios. For this, many students used Artificial Intelligence (AI) chatbots employing Large Language Models (LLMs). This offered a natural quasi-experiment in a real-life learning setting to compare students’ DOM-COR performance with and without LLMs. Students also reported on LLM-tools use for study and everyday purposes as well as on their learning outcomes (e.g., credit points, exams). This study investigates both the relationships between LLM-tools use and students DOM-COR assessment performance as well as their learning outcomes at universities. We explicitly distinguish between the qualitative indicators of learning and the speed thereof in the context of LLM-chatbots use. By comparing the time students spent completing DOM-COR tasks and their performance between LLM-users and non-users, we examine if, and to what extent, LLM-users had an advantage. While LLM-users were faster, they did not perform better on written task responses. The relationships with learning outcomes trend in the same direction, indicating that regular LLM-users passed more exams, but did not appear to achieve better grades. The findings point to opportunities and risks when using LLM for domain-specific COR-like tasks as well as for regular study assignments.