Conventionally Developed Versus Large Language Model-Generated Exam Tasks in Higher Education: A Comparative Analysis

This exploratory field study investigates how human-screened exam tasks generated by the large language model (LLM) GPT-4o perform in comparison with conventionally developed exam tasks with regard to selected quality criteria. Using two business administration modules at a German distance-learning university, human-screened LLM-generated tasks were integrated into submission assignments (SA) alongside instructor-created tasks. Four selected task-quality indicators were examined: objective difficulty, subjective difficulty, context relevance, and item discrimination. The analysis combines descriptive statistics with paired-sample t-tests and Wilcoxon signed-rank tests, applied at three aggregation levels: SA-specific, module-level, and cross-module. The results revealed mixed and task-specific patterns. While selected indicators were numerically similar for some task pairs, statistically significant and, in one comparison, substantial differences were observed in others, particularly in objective difficulty. Given that the study examined only four task pairs that were not systematically matched and that prior human screening was conducted, the findings establish neither equivalence nor general comparability. However, the study provides initial exploratory insights from two business administration modules that can serve as a foundation for further research.