Curriculum-aligned analysis of assessment feedback: a retrieval-augmented large language model approach for revealing fine-grained student error patterns in higher education

Abstract Assessment feedback in higher education provides fine-grained evidence about students’ specific errors but remains underutilized as a data source for teaching improvement. This study proposes a method based on a large language model (LLM) that transforms unstructured feedback into curriculum-aligned error analysis in three steps, deriving an error taxonomy from the feedback, mapping each piece to curriculum content by retrieval-augmented generation, and aggregating the annotations into error patterns with quantified cohort coverage. The method was applied to 6072 pieces of feedback across four cohorts (2022–2025) of an undergraduate engineering course. The feedback resolved into recurring, curriculum-aligned error patterns, dominated by missing content and justification, clustered in identifiable modules and weeks, and more than half recurring across cohorts. The resulting annotations achieved 89.1% agreement with human judgment, substantially higher than the 74.7% obtained when an expert-designed taxonomy was applied to the same data. In a focus group, the teaching team confirmed the patterns against their marking experience, proposed course revisions from them, and set the bound that they serve as a reference and not as a direct measure of teaching quality. The study reports a process account of this team’s use of curriculum-aligned error analytics, built on sensemaking and pedagogical response. The analysis turns feedback written for students into evidence on the teaching that preceded it, by which a teaching team can check the alignment between intended outcomes, teaching, and assessment in students’ actual work, reflect on its own practice, and improve its teaching across cohorts.

Authors

Institutions

Publication Details

Journal
International Journal of Educational Technology in Higher Education
Published
2026-11-15
DOI
https://doi.org/10.1186/s41239-026-00640-7
Primary Topic
Online Learning and Analytics
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Curriculum-aligned analysis of assessment feedback: a retrieval-augmented large language model approach for revealing fine-grained student error patterns in higher education

Qingtan Shen, Xu Jia, Lirui Guo, Linxin Hua et al.
1 links
International Journal of Educational Technology in Higher Education
Online Learning and Analytics
article

Curriculum-aligned analysis of assessment feedback: a retrieval-augmented large language model approach for revealing fine-grained student error patterns in higher education

Qingtan Shen, Xu Jia, Lirui Guo, Linxin Hua, Nan Zheng, Ye Lu, Dilang Tan
article en

Abstract

Abstract Assessment feedback in higher education provides fine-grained evidence about students’ specific errors but remains underutilized as a data source for teaching improvement. This study proposes a method based on a large language model (LLM) that transforms unstructured feedback into curriculum-aligned error analysis in three steps, deriving an error taxonomy from the feedback, mapping each piece to curriculum content by retrieval-augmented generation, and aggregating the annotations into error patterns with quantified cohort coverage. The method was applied to 6072 pieces of feedback across four cohorts (2022–2025) of an undergraduate engineering course. The feedback resolved into recurring, curriculum-aligned error patterns, dominated by missing content and justification, clustered in identifiable modules and weeks, and more than half recurring across cohorts. The resulting annotations achieved 89.1% agreement with human judgment, substantially higher than the 74.7% obtained when an expert-designed taxonomy was applied to the same data. In a focus group, the teaching team confirmed the patterns against their marking experience, proposed course revisions from them, and set the bound that they serve as a reference and not as a direct measure of teaching quality. The study reports a process account of this team’s use of curriculum-aligned error analytics, built on sensemaking and pedagogical response. The analysis turns feedback written for students into evidence on the teaching that preceded it, by which a teaching team can check the alignment between intended outcomes, teaching, and assessment in students’ actual work, reflect on its own practice, and improve its teaching across cohorts.

International Journal of Educational Technology in Higher EducationVol. 23(1)
Deakin University (AU), The University of Melbourne (AU), Monash University (AU)
Quality education
Online Learning and Analytics
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon