A Teacher-Centered Evaluation of AI-Assisted Educational Assessment: Rubric Consistency, Feedback Quality, and Fairness Across Student Groups

AI-assisted assessment is increasingly used to score student writing and to generate formative feedback, yet it is still validated mainly by aggregate agreement with consensus scores. For classroom adoption the decisive question is teacher-centered: does the AI apply the rubric the way a competent teacher does, diagnose the same weaknesses, and treat every student group alike? We propose a Teacher-Centered Evaluation (TCE) framework with four lenses—teacher-relative rubric consistency, rubric-guided counterfactual tests, feedback diagnostic validity, and score-conditional fairness—complemented by a teacher-in-the-loop routing policy with group-audited conformal intervals. TCE is instantiated on two public corpora released by the research team behind the ASAP 2.0 benchmark: PERSUADE 2.0 (5,531 argumentative essays with holistic scores, 55,068 expert-rated discourse elements, and student demographics) and ELLIPSE (6,462 essays double-scored by 27 trained raters on seven rubric traits). Five AI scoring engines are audited. The strongest engine reaches a quadratic weighted kappa (QWK) of 0.870 and, in a teacher-substitution test, agrees with a teacher almost exactly as well as a second teacher does (ΔQWK = −0.016). TCE nevertheless exposes failures that agreement statistics hide: scores are blind to sentence order, rise by 0.27 points when a paragraph is duplicated, and fall no more when evidence is deleted than when equally long non-evidence text is removed; analytic trait scores collapse into a single dimension (inter-trait r = 0.87 vs. 0.61 for teachers); feedback misses 70% of the weaknesses that experts flag in English-learner writing; and conditioning on the teacher’s score uncovers subgroup gaps that raw statistics mask. Routing 20% of essays to teachers with a learned deferral model raises QWK to 0.907, reduces the worst subgroup bias by 31%, and nearly halves the English-learner agreement gap.

Authors

Institutions

Publication Details

Journal
Academic journal of management and social sciences
Published
2026-11-18
DOI
https://doi.org/10.54097/g70yrm22
Primary Topic
Artificial Intelligence in Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

A Teacher-Centered Evaluation of AI-Assisted Educational Assessment: Rubric Consistency, Feedback Quality, and Fairness Across Student Groups

Qianyi Fang
Academic journal of management and social sciences
Artificial Intelligence in Education
article

A Teacher-Centered Evaluation of AI-Assisted Educational Assessment: Rubric Consistency, Feedback Quality, and Fairness Across Student Groups

Qianyi Fang
article en

Abstract

AI-assisted assessment is increasingly used to score student writing and to generate formative feedback, yet it is still validated mainly by aggregate agreement with consensus scores. For classroom adoption the decisive question is teacher-centered: does the AI apply the rubric the way a competent teacher does, diagnose the same weaknesses, and treat every student group alike? We propose a Teacher-Centered Evaluation (TCE) framework with four lenses—teacher-relative rubric consistency, rubric-guided counterfactual tests, feedback diagnostic validity, and score-conditional fairness—complemented by a teacher-in-the-loop routing policy with group-audited conformal intervals. TCE is instantiated on two public corpora released by the research team behind the ASAP 2.0 benchmark: PERSUADE 2.0 (5,531 argumentative essays with holistic scores, 55,068 expert-rated discourse elements, and student demographics) and ELLIPSE (6,462 essays double-scored by 27 trained raters on seven rubric traits). Five AI scoring engines are audited. The strongest engine reaches a quadratic weighted kappa (QWK) of 0.870 and, in a teacher-substitution test, agrees with a teacher almost exactly as well as a second teacher does (ΔQWK = −0.016). TCE nevertheless exposes failures that agreement statistics hide: scores are blind to sentence order, rise by 0.27 points when a paragraph is duplicated, and fall no more when evidence is deleted than when equally long non-evidence text is removed; analytic trait scores collapse into a single dimension (inter-trait r = 0.87 vs. 0.61 for teachers); feedback misses 70% of the weaknesses that experts flag in English-learner writing; and conditioning on the teacher’s score uncovers subgroup gaps that raw statistics mask. Routing 20% of essays to teachers with a learned deferral model raises QWK to 0.907, reduces the worst subgroup bias by 31%, and nearly halves the English-learner agreement gap.

Academic journal of management and social sciencesVol. 18(2)
Johns Hopkins University (US)
Artificial Intelligence in Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon