The Illusion of Learning Resulting from LLM-Generated Multiple-Choice Questions in Video-Based Learning

Many educators commonly use embedded Multiple-Choice Questions (MCQs) to support learning from videos. However, developing well-crafted MCQs is a time-consuming task for educators. This study investigates the effectiveness of Large Language Model (LLM)-generated MCQs compared to MCQs written by a subject matter expert (SME) in supporting student learning gains. In an experimental design, participants watched an educational video and were exposed to either LLM-generated MCQs or MCQs written by an SME. Learning gains were assessed by comparing pre-test results with immediate and delayed post-test results to measure short-term and long-term knowledge retention. Results reveal an illusion of learning: participants who received LLM-generated MCQs answered 86% of those questions correctly during the video, yet the control group, which received no questions, significantly outperformed the LLM group on the immediate post-test. However, in comparison with the delayed post-test, no significant group differences were identified, indicating that MCQs, whether LLM-generated or SME-written, might not support learning at all. Although higher in-video MCQ performance significantly predicted higher delayed retention in the SME condition but did not in the LLM condition. SME-written MCQs supported a coherent learning trajectory in which in-video MCQ scores related to immediate post-test scores, and immediate post-test scores related to delayed post-test scores. The findings suggest that in-video MCQs may not support learning, whereas SME-written MCQs support a consistent learning trajectory.