ChatGPT In Assessment: A Comparison of Artificial Intelligence (AI)-Generated and Human-Generated One Best-Type Multiple Choice Questions

Objective: To compare the quality of Multiple-Choice Questions (MCQs) created by humans and Chat GPT using post-hoc item analysis. Study Design: Quasi-Experimental Research Design. Place and Duration of Study: Department of Medicine, Azra Naheed Medical College, The Superior University, Lahore Pakistan, from Feb to Aug 2024. Methodology: A total of 100 participants were given two sets of 50 questions. One set was created using Chat GPT while the other was crafted by human teachers. Duration of generation of MCQs was noted. Post hoc item analysis was done (difficulty index, discrimination index and distractor analysis). Student scores were compared. Independent sample t-test was applied to compare mean difficulty index, discrimination index and student scores. Two proportions z-test to compare mean distractor efficiency. The results were reported as means, frequencies and percentages and analyzed using SPSS version 27. Results: Total of 100 MCQs were evaluated. Chat GPT generated MCQs in 25 minutes, significantly faster than humans' 205 minutes. The Difficulty Index (Chat GPT: 30.3, humans: 34.7, p=0.39), Discrimination Index (Chat GPT: 0.21, humans: 0.19, p=0.514), student scores (Chat GPT: 19.4, humans: 18.46, p=0.125) and Distractor efficiency (Chat GPT: 75%, humans 69.5%, p=0.385) were statistically non-significant, indicating similar quality. Conclusion: The quality of MCQs produced by Chat GPT and Human Educators is comparable; however, the duration required by Chat GPT is significantly less. A mixed approach, where subject matter experts qualitatively evaluate the MCQs generated by Chat GPT, can enhance the overall assessment quality.