Comparative evaluation of ChatGPT-5 and DeepSeek-V3 for generating prosthodontic multiple-choice questions: a cross-sectional study based on blinded expert ratings

Although manual development of high-quality multiple-choice questions (MCQs) is time-consuming, MCQs continue to play a vital role in knowledge assessment for dental education. In prosthodontics, item development is particularly demanding as it requires broad theoretical coverage and precise clinical knowledge. Evidence comparing the quality of MCQs generated by different large language models (LLMs) remains limited. This study compared the content validity and reliability of prosthodontic MCQs generated by ChatGPT-5 and DeepSeek-V3. Each model generated 100 prosthodontic MCQs based on all five chapters of the 8th edition of Prosthodontics. Two prosthodontic experts independently rated all items on six dimensions using a five-point scale: grammatical clarity and accuracy, non-misleading stem, distractor reasonableness, discrimination, answer correctness, and relevance to the textbook. A third senior adjudicating expert classified each item as suitable for direct examination use, suitable for use after revision, or unsuitable for examination use. Descriptive statistics, intraclass correlation coefficients, and Mann-Whitney U tests were employed to evaluate and compare the performance of the two models. Expert ratings for items generated by both models demonstrated moderate-to-excellent inter-rater reliability, indicating a reasonably stable evaluation process. Overall, both ChatGPT-5 and DeepSeek-V3 produced prosthodontic MCQs of acceptable quality, with comparable mean scores. ChatGPT-5 exhibited superior performance in grammatical clarity and accuracy as well as distractor reasonableness, whereas DeepSeek-V3 scored slightly higher in relevance to the textbook. However, both models also generated flawed items, including ambiguous stems, trivial or multiple-keyed options, limited discrimination, and occasional factual inaccuracies. Some questions required substantial revision or exclusion prior to examination use. ChatGPT-5 and DeepSeek-V3 efficiently generated prosthodontic MCQs with similar overall quality and distinct individual strengths. LLMs may offer valuable support for assessment design in dental education, nevertheless, expert review remains essential to ensure content accuracy, educational quality, and assessment fairness. An artificial intelligence-human collaborative workflow can therefore serve as an effective strategy for MCQ development.