Generative AI vs Student Performance on Multiple-Choice, General Chemistry Exams: Insights into Question Design from Regression Analysis

Generative artificial intelligence (AI) is and will continue to be an impactful resource in chemical education. Contrasting numerous positive applications are the concerns about its use for cheating. Regarding the latter, it has been demonstrated that generative AI usage can be detected, even on multiple-choice chemistry exams. But this raises the questions “Why is there a performance difference?” and “How does ChatGPT ‘think’ differently than students?” Here, binomial regression analysis was performed on student and ChatGPT question (i.e., item) success rates on multiple-choice general chemistry II exams with respect to item variables including: format/components (e.g., containing “not”, fill in the blank, capitalization, etc.), topical content (i.e., book sections/chapters), knowledge type, cognitive process, etc. There were some similar trends in performance; for example, every doubling of the word count resulted in a 10–20% decrease in the odds of both ChatGPT and students answering the question correctly. There were also contrasting outcomes, like with questions containing “not,” where ChatGPT was much more likely to answer correctly as compared with students. On the other hand, students performed better on questions requiring interpretation and classification subcategories of the “understanding” knowledge type. Given the rapid evolution and diversity of generative AI models, the specific outcomes of this analysis will quickly become outdated, but this report demonstrates item variable categorization and regression analysis as an effective tool for highlighting the performance differences between generative AI and students. Furthermore, these insights may help guide item design where the goal is to mitigate the impacts of cheating via generative AI usage or in designing online surveys where obtaining meaningful results requires one to differentiate human and generative AI responses.