Quasi-experimental evaluation of a human-centric AI tutoring and human-in-the-loop assessment system in advanced STEM education

Artificial intelligence is reshaping higher education, yet most deployments of large-language-model tutoring and automated grading target introductory courses and pay limited attention to the transparency, trust, and equity concerns that become acute in advanced, safety-critical technical domains. This paper presents a human-centric AI-assisted instructional system and evaluates it in a postgraduate hardware-security course on electromagnetic protection and side-channel analysis. The system couples three human-centric components: (i) a retrieval-augmented tutoring assistant whose answers are grounded in citable course materials to curb hallucination and preserve traceability; (ii) a human-in-the-loop assessment pipeline in which a language model drafts rubric-based feedback that instructors verify item by item; and (iii) an adaptive, tiered project-based learning path that offers differentiated entry points. We evaluated the system in a quasi-experiment with 47 postgraduates (24 experimental, 23 control) taught concurrently by the same instructors in parallel class sections. The experimental group’s competency score rose from 62.4 to 84.7, against 61.8 to 70.2 for the control group, a difference in gains that was statistically significant and associated with a large effect size ( \(t(41.2)=4.55\) , \(p<0.001\) , Cohen’s \(d=1.32\) ), while its score dispersion contracted from a standard deviation of 14.5 to 7.2. A baseline-by-group interaction indicated significantly stronger differential improvement among initially lower-scoring students in the experimental section, although this distributional finding is interpreted cautiously given the non-randomised design. Human-in-the-loop grading was associated with a 35.7% reduction in marking time while reaching an AI–instructor agreement of quadratic-weighted Cohen’s \(\kappa =0.95\) ; because this agreement measures consistency rather than correctness, we interpret it accordingly. We examine how human oversight, citation grounding, and adaptivity may have jointly contributed to these outcomes, treating the intervention as an integrated bundle whose individual components remain to be isolated, and we set out the responsible-AI safeguards that such deployments require.

Authors

Institutions

Publication Details

Journal
Scientific Reports
Published
2026-11-16
DOI
https://doi.org/10.1038/s41598-026-79651-3
Primary Topic
Intelligent Tutoring Systems and Adaptive Learning
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Quasi-experimental evaluation of a human-centric AI tutoring and human-in-the-loop assessment system in advanced STEM education

Chaoen Xiao, Wang Jianxin, Runze Zhou, Zixuan Wang et al.
Scientific Reports
Intelligent Tutoring Systems and Adaptive Learning
article

Quasi-experimental evaluation of a human-centric AI tutoring and human-in-the-loop assessment system in advanced STEM education

Chaoen Xiao, Wang Jianxin, Runze Zhou, Zixuan Wang, Lei Zhang
article en

Abstract

Artificial intelligence is reshaping higher education, yet most deployments of large-language-model tutoring and automated grading target introductory courses and pay limited attention to the transparency, trust, and equity concerns that become acute in advanced, safety-critical technical domains. This paper presents a human-centric AI-assisted instructional system and evaluates it in a postgraduate hardware-security course on electromagnetic protection and side-channel analysis. The system couples three human-centric components: (i) a retrieval-augmented tutoring assistant whose answers are grounded in citable course materials to curb hallucination and preserve traceability; (ii) a human-in-the-loop assessment pipeline in which a language model drafts rubric-based feedback that instructors verify item by item; and (iii) an adaptive, tiered project-based learning path that offers differentiated entry points. We evaluated the system in a quasi-experiment with 47 postgraduates (24 experimental, 23 control) taught concurrently by the same instructors in parallel class sections. The experimental group’s competency score rose from 62.4 to 84.7, against 61.8 to 70.2 for the control group, a difference in gains that was statistically significant and associated with a large effect size ( \(t(41.2)=4.55\) , \(p<0.001\) , Cohen’s \(d=1.32\) ), while its score dispersion contracted from a standard deviation of 14.5 to 7.2. A baseline-by-group interaction indicated significantly stronger differential improvement among initially lower-scoring students in the experimental section, although this distributional finding is interpreted cautiously given the non-randomised design. Human-in-the-loop grading was associated with a 35.7% reduction in marking time while reaching an AI–instructor agreement of quadratic-weighted Cohen’s \(\kappa =0.95\) ; because this agreement measures consistency rather than correctness, we interpret it accordingly. We examine how human oversight, citation grounding, and adaptivity may have jointly contributed to these outcomes, treating the intervention as an integrated bundle whose individual components remain to be isolated, and we set out the responsible-AI safeguards that such deployments require.

Scientific Reports
Beijing Electronic Science and Technology Institute (CN)
Chinese Academy of Sciences
Quality education
Intelligent Tutoring Systems and Adaptive Learning
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon