Prompt Design and Automated Assessment: A Comparative Evaluation of Claude and ChatGPT Against Teacher Marking of Handwritten Examinations
This study compares Claude and ChatGPT against teacher marking of handwritten Psychology examination scripts, examining how prompt design affects large language models (LLMs)–human agreement. Sixty-nine handwritten scripts, as photographic images, were graded by both LLMs under four prompting strategies (minimal, rubric, exemplar, role-based). Both LLMs differed significantly from the teacher in seven of eight conditions; Claude under role-based prompting was the sole exception (p = 0.038). ChatGPT overmarked across all four prompts, while Claude’s deviation varied across prompt conditions. Prompt type had a significant impact on grading accuracy with a significant LLM × Prompt interaction (p = 0.001, η²p = 0.08). Claude under the exemplar prompt had the closest agreement with the teacher’s marks (MAE = 2.42; r = 0.84). A grade-level analysis showed that scoring accuracy did not always result in grade-level agreement, indicating both metrics are needed. Prompt design improved LLM grading accuracy, but neither model reproduced teachers’ marks closely enough to replace human oversight in high-stakes assessment.
Authors
- Ritu Dangwal (ORCID: https://orcid.org/0000-0002-6950-1108)
- Siddharth Gurnani
Institutions
- NIIT University (IN)
- Systems, Applications & Products in Data Processing (Germany) (DE)
Publication Details
- Journal
- The International Journal of Assessment and Evaluation
- Published
- 2026-11-20
- DOI
- https://doi.org/10.18848/2327-7920/cgp/a190
- Primary Topic
- Artificial Intelligence in Education
- Type
- article
- Field-Weighted Citation Impact
- 0.00