Automatic Multiple Choice Question Generation: A Systematic Literature Review

Despite their drawbacks, multiple-choice questions (MCQ) have been widely used to assess the students' understanding of lectures through examinations. The development of automatic MCQ generation is beneficial, especially for educators. As a starting point for further development, a Systematic Literature Review (SLR) is conducted to uncover current trends, future challenges, and opportunities in automatic MCQ generation. Previously, an SLR was conducted, but it lacks coverage of the utilization of transformer-based models. This SLR covers the development of automatic MCQ generation using either traditional or advanced approaches such as Transformers. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses framework was used to gather the data from Scopus, IEEE Xplore, SpringerLink, arXiv, and Semantic Scholar. The included articles must be open-access computer science conference papers or journal articles and written in English less than five years ago. Four independent reviewers analyzed the research workflow, evaluation metric, and dataset used in each study. There are 18 included studies, where 17% (n = 3) studies are from 2024, 33% (n = 6) studies are from 2023, 22% (n = 4) studies are from 2022, 11% (n = 2) studies are from 2021, and 17% (n = 3) studies are from 2020. There are 33% (n = 6) of the studies used the traditional feature-based engineering approach, 39% (n = 7) of the studies used the Transformer-based model fine-tuning approach, and the remaining used novel approaches. The study found that BERT variants are the most utilized Transformer-based model in automatic MCQ. The research notes some challenges, but also open various opportunities for further research, including Large Language Model (LLM) utilization for automatic MCQ generation, the utilization BERT-based models for standardized machine-learned evaluation metrics, and the initiative for the creation of an MCQ dataset benchmark.