A Study on the Validity of Automatically Generated Questions Using the Q-Craft

Objectives In an environment where the demand for automatic item generation is increasing, it is crucial for automatically generated items to match the quality of manually created items to be effectively utilized in educational settings. This study aims to automatically generate items (AIG items) similar to traditional manually generated items (MIG items) using an automatic item generation program and to evaluate the validity of these automatically generated items by analyzing their similarity to MIG items. Methods 15 representative items types from the March 2023 National Assessment of Education for 12th grade, administered by the Seoul Metropolitan Office of Education, were selected to form the MIG items. Similarly, 15 AIG items were generated using the Q-Craft automatic item generation program. The AIG items were subjected to blind review by 10 subject-matter experts and were simultaneously administered along with the MIG items to 48 first-year university students. Results The expert review of the AIG items indicated a high level of naturalness in the flow of the passages and the use of English expressions. However, the attractiveness of the answer choices received relatively lower ratings. The analysis of the responses from the university students revealed a high degree of similarity between the MIG and AIG items, with a parallel test reliability of .662. Furthermore, the comparison using test information functions showed that the ability levels of students providing the maximum test information were identical for both sets of items. Conclusions The AIG items are not yet fully equivalent to the MIG items. Nonetheless, the fact that the AIG items accurately estimated the ability levels of students, as the MIG items did, is highly significant. This suggests that AIG items can be deemed valid depending on the testing purpose.