A comparative pilot study of the accuracy consistency and marking scheme responsiveness of AI chatbots and human evaluators in professional accounting assessment

This pilot study investigates whether artificial intelligence (AI) can effectively assess professional accounting examination scripts comparably to human expert evaluation. The research examines Claude 3.5, GPT-4o, Perplexity, and DeepSeek-V2 across nine subject areas of Ghanaian professional accounting examinations administered by the Institute of Chartered Accountants, Ghana (ICAG). A quantitative experimental design analysed 27 scripts across foundation, application, and professional levels, with each script assessed three times per condition per model, yielding 216 AI-based assessments for comparison with human examiner marks. Statistical analyses included mean absolute deviation (MAD), paired t-tests, chi-square and F-tests, and a novel Marking Scheme Responsiveness Index (MSRI) introduced to quantify AI responsiveness to structured criteria. Claude 3.5 demonstrated the closest alignment with the reference examiner’s marks under guided conditions, achieving a MAD of 4.1 points and a 46.1% improvement over its unguided condition (Cohen’s d = 1.12, 95% CI [0.52, 1.72]); however, as a single uniform prompt was applied across all models, this comparative advantage should be interpreted with caution given the well-documented sensitivity of large language model performance to prompt design. Conversely, GPT-4o showed deteriorating alignment under guided conditions, a counterintuitive finding that warrants further investigation. It is important to note that all AI scores are benchmarked against the marks of a single reference examiner, and the absence of human inter-examiner reliability data limits the interpretability of AI-human deviations. Results suggest assessment structure and model selection may be more influential than subject complexity in determining AI-human score alignment. Four contributions are made: the first empirical benchmark of four state-of-the-art large language models (LLMs) in professional accounting credentialing; operationalisation of the MSRI as a reusable metric; hypotheses challenging prevailing assumptions about AI in professional credentialing assessment; and a validated protocol for future large-scale investigation. All findings are hypothesis-generating, with confidence intervals reported throughout to guide replication planning.