Reliability of ChatGPT in scoring medical undergraduates’ portfolio in professionalism

Artificial Intelligence tools have assumed an important role in medical education including assessment. Evidence supports its capabilities in automated grading/scoring processes in different types of written assessments. To estimate the level of agreement between scores given by the lecturers and GPT-5 for reflectively written portfolios by students for a formative assessment in professionalism in medical education. A proof-of-concept study was conducted to score 156 previously scored portfolios by lecturers with adherence to a rubric. Each portfolio was scored by GPT-5, using the same detailed rubric integrated into a structured prompt designed with a prompt engineering framework. Intraclass Correlation Coefficient (ICC) showed ‘good’ level of agreement (ICC (A,1) = 0.816) between scores given by lecturers and GPT-5, with a strong linear association [Pearson r = 0.82 ( p < 0.001) and Spearman ρ = 0.80 ( p < 0.001)]. Same level of ICC agreement and strong linear association were observed for individual lecturer’s scores and corresponding GPT-5 scores. Also, ICC showed ‘excellent’ level of agreement (ICC (C,1) = 0.959) between scores given for the same portfolio by GPT-5 in three cycles, with ‘excellent’ internal consistency (Cronbach α = 0.9862). GPT-5 demonstrated good agreement with lecturer’s scores and has potential for complementing the reflectively written assessment scoring process in medical education. Its capacity to further augment the scoring process with minimal human intervention should be further investigated.