Identifying Features of LLM-Resistant Exam Questions: Insights from Artificial Intelligence (AI)–Student Performance Comparisons

Large language models (LLMs) are rapidly being explored as tools to support learning and assessment in health science education, yet their performance across discipline-specific evaluations remains underexamined. This study evaluated the accuracy of two prominent LLMs on university-level pharmacognosy examinations and compared their performance to that of pharmacy students. Authentic exam papers comprising a range of question formats and content categories were administered to ChatGPT and DeepSeek using a structured prompting approach. Student data were anonymized and LLM responses were graded using the same marking criteria applied to student cohorts, and a Monte Carlo simulation was conducted to determine whether observed performance differences were statistically meaningful. Facility Index (FI) values were calculated to contextualize item difficulty and identify where LLM performance aligned or diverged from student outcomes. The models demonstrated variable accuracy across question types, with a stronger performance in recall-based and definition-style items and comparatively weaker outputs for applied or interpretive questions. Simulated comparisons showed that LLM performance did not uniformly exceed or fall below that of students, indicating dimension-specific strengths and constraints. These findings suggest that while LLM-resistant examination design is contingent on question structure and content, further research should refine their integration into pharmacy education.