A Study on Automatic Item Generation for Multiple-Choice Reading Comprehension Assessment in Korean Language

This study investigates the potential of generative AI-based Automatic Item Generation (AIG) technology for creating Korean language reading comprehension multiple-choice items. Three prompt strategies—Zero-shot, Few-shot, and Chain- of-Thought (CoT)—were analyzed, with items generated by each strategy evaluated by a panel of eight experts (five Korean language teachers and three doctoral researchers) based on educational validity, format appropriateness, and content fidelity. The CoT prompt demonstrated superior performance by guiding AI through step-by-step reasoning, effectively generating items that balanced comprehension levels from factual understanding to creative and evaluative reasoning while maintaining logical coherence and alignment with assessment objectives. The Few-shot prompt achieved structural completeness but lacked creativity and diversity in certain item types. The Zero-shot prompt primarily focused on basic information reconstruction, proving inadequate for assessing higher-order thinking skills. These findings suggest that generative AI technology can serve as a valuable supplementary tool for teachers’ expertise in assessment design, particularly in enhancing the systematic development and validity of test items. The CoT prompt specifically showed promise in reducing teachers’ workload while maintaining assessment quality. This study highlights the potential of AIG technology as a practical tool for improving educational assessment quality and supporting teachers’ professional practice in item development.