Protocol for Developing Constructed-Response Items Using ChatGPT and Quality Evaluation: Focusing on Korean Language Arts

This study explores prompt guidelines for generating constructed-response items using ChatGPT and examines the quality of the AI-generated items in comparison to human-developed items. Based on a literature review and testing of various prompting strategies, we developed a three-step prompt guideline for generating both constructed-response items and scoring rubrics. We then compared the quality of the ChatGPT-generated items to human-developed items across 20 evaluation criteria. The results showed that the quality of the ChatGPT-generated items was significantly lower than that of the human-developed items, particularly in terms of task clarity, alignment with reading materials, compliance with the national curriculum, and reliability of scoring criteria. However, no significant difference was observed between the ChatGPT and human-developed items in areas such as alignment among learning objectives, task content, scoring criteria, and vocabulary appropriateness. Finally, implications for using AI in item generation and designing human-AI collaborative assessments were discussed.