Exploring Methods for Evaluating Scientific Drawings Using Artificial Intelligence

Objectives This study aims to explore the potential of utilizing artificial intelligence (AI) in evaluating scientific drawings and to derive implications for the application of AI in this context. Methods For this study, assessment items and criteria for evaluating scientific drawings were developed and administered to 40 sixth-grade elementary school students. The students' responses were categorized by type and analyzed accordingly. Five teachers evaluated the students' responses, and an artificial intelligence model conducted five rounds of evaluation on the same responses. A custom GPT model, specifically developed for scientific drawing assessment, was applied as the AI evaluator. To analyze the agreement between the evaluations made by teachers and the AI, Fleiss' Kappa and Cohen's Kappa coefficients were calculated and compared. Additionally, a survey was conducted with the teachers to collect their opinions on the AI-generated evaluation results. Results When student responses were categorized as non-responsive, the evaluation results of both teachers and artificial intelligence (AI) were in complete agreement for all three items. In the text-based response type, the evaluation results of teachers and AI showed complete agreement in two out of three items. For the image-based and combined image-and-text response types, the evaluation results demonstrated very weak agreement between teachers and AI. Overall, teachers generally agreed with the feedback generated by the AI based on its evaluation of student responses. Conclusions The findings suggest that while it is currently impractical to entirely delegate the evaluation of scientific drawings to AI, this study provides insights into strategies for integrating AI into the evaluation process.