Prompt Design and Automated Assessment: A Comparative Evaluation of Claude and ChatGPT Against Teacher Marking of Handwritten Examinations

This study compares Claude and ChatGPT against teacher marking of handwritten Psychology examination scripts, examining how prompt design affects large language models (LLMs)–human agreement. Sixty-nine handwritten scripts, as photographic images, were graded by both LLMs under four prompting strategies (minimal, rubric, exemplar, role-based). Both LLMs differed significantly from the teacher in seven of eight conditions; Claude under role-based prompting was the sole exception (p = 0.038). ChatGPT overmarked across all four prompts, while Claude’s deviation varied across prompt conditions. Prompt type had a significant impact on grading accuracy with a significant LLM × Prompt interaction (p = 0.001, η²p = 0.08). Claude under the exemplar prompt had the closest agreement with the teacher’s marks (MAE = 2.42; r = 0.84). A grade-level analysis showed that scoring accuracy did not always result in grade-level agreement, indicating both metrics are needed. Prompt design improved LLM grading accuracy, but neither model reproduced teachers’ marks closely enough to replace human oversight in high-stakes assessment.

Authors

Institutions

Publication Details

Journal
The International Journal of Assessment and Evaluation
Published
2026-11-20
DOI
https://doi.org/10.18848/2327-7920/cgp/a190
Primary Topic
Artificial Intelligence in Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Prompt Design and Automated Assessment: A Comparative Evaluation of Claude and ChatGPT Against Teacher Marking of Handwritten Examinations

Ritu Dangwal, Siddharth Gurnani
3 links
The International Journal of Assessment and Evaluation
Artificial Intelligence in Education
article

Prompt Design and Automated Assessment: A Comparative Evaluation of Claude and ChatGPT Against Teacher Marking of Handwritten Examinations

Ritu Dangwal, Siddharth Gurnani
article en

Abstract

This study compares Claude and ChatGPT against teacher marking of handwritten Psychology examination scripts, examining how prompt design affects large language models (LLMs)–human agreement. Sixty-nine handwritten scripts, as photographic images, were graded by both LLMs under four prompting strategies (minimal, rubric, exemplar, role-based). Both LLMs differed significantly from the teacher in seven of eight conditions; Claude under role-based prompting was the sole exception (p = 0.038). ChatGPT overmarked across all four prompts, while Claude’s deviation varied across prompt conditions. Prompt type had a significant impact on grading accuracy with a significant LLM × Prompt interaction (p = 0.001, η²p = 0.08). Claude under the exemplar prompt had the closest agreement with the teacher’s marks (MAE = 2.42; r = 0.84). A grade-level analysis showed that scoring accuracy did not always result in grade-level agreement, indicating both metrics are needed. Prompt design improved LLM grading accuracy, but neither model reproduced teachers’ marks closely enough to replace human oversight in high-stakes assessment.

The International Journal of Assessment and Evaluation
NIIT University (IN), Systems, Applications & Products in Data Processing (Germany) (DE)
Quality education
Artificial Intelligence in Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon