Quasi-experimental evaluation of a human-centric AI tutoring and human-in-the-loop assessment system in advanced STEM education
Artificial intelligence is reshaping higher education, yet most deployments of large-language-model tutoring and automated grading target introductory courses and pay limited attention to the transparency, trust, and equity concerns that become acute in advanced, safety-critical technical domains. This paper presents a human-centric AI-assisted instructional system and evaluates it in a postgraduate hardware-security course on electromagnetic protection and side-channel analysis. The system couples three human-centric components: (i) a retrieval-augmented tutoring assistant whose answers are grounded in citable course materials to curb hallucination and preserve traceability; (ii) a human-in-the-loop assessment pipeline in which a language model drafts rubric-based feedback that instructors verify item by item; and (iii) an adaptive, tiered project-based learning path that offers differentiated entry points. We evaluated the system in a quasi-experiment with 47 postgraduates (24 experimental, 23 control) taught concurrently by the same instructors in parallel class sections. The experimental group’s competency score rose from 62.4 to 84.7, against 61.8 to 70.2 for the control group, a difference in gains that was statistically significant and associated with a large effect size ( \(t(41.2)=4.55\) , \(p<0.001\) , Cohen’s \(d=1.32\) ), while its score dispersion contracted from a standard deviation of 14.5 to 7.2. A baseline-by-group interaction indicated significantly stronger differential improvement among initially lower-scoring students in the experimental section, although this distributional finding is interpreted cautiously given the non-randomised design. Human-in-the-loop grading was associated with a 35.7% reduction in marking time while reaching an AI–instructor agreement of quadratic-weighted Cohen’s \(\kappa =0.95\) ; because this agreement measures consistency rather than correctness, we interpret it accordingly. We examine how human oversight, citation grounding, and adaptivity may have jointly contributed to these outcomes, treating the intervention as an integrated bundle whose individual components remain to be isolated, and we set out the responsible-AI safeguards that such deployments require.
Authors
- Chaoen Xiao (ORCID: https://orcid.org/0000-0001-6885-9063)
- Wang Jianxin
- Runze Zhou
- Zixuan Wang (ORCID: https://orcid.org/0009-0005-9166-1692)
- Lei Zhang
Institutions
- Beijing Electronic Science and Technology Institute (CN)
Publication Details
- Journal
- Scientific Reports
- Published
- 2026-11-16
- DOI
- https://doi.org/10.1038/s41598-026-79651-3
- Primary Topic
- Intelligent Tutoring Systems and Adaptive Learning
- Type
- article
- Field-Weighted Citation Impact
- 0.00
Funders
- Chinese Academy of Sciences