DiaBench: A Benchmarking Framework to Evaluate the Performance of LLMs in T2D Patient Education

Background Effective type 2 diabetes (T2D) education improves patient outcomes, but clinician time constraints limit what can be covered during encounters. Large language models (LLMs) offer a scalable way to answer patient questions, yet their accuracy and reliability in T2D education across health literacy levels remain poorly characterized. Methods We developed DiaBench, a standardized benchmarking framework built from 96 question-and-answer pairs collected by clinicians from representative real-world patient encounters. GPT-5 transformed each pair into multiple-choice questions (MCQs) under three configurations (1-3) simulating different question formats and levels of user background. A five-member panel of certified diabetes care and education specialists (CDCESs), medical doctors (MDs), and endocrinologists validated the transformations, rating question rephrasing, answer rephrasing, and answer correctness. We then evaluated seven LLMs (GPT-51, Gemini-2.5-Flash2, Cohere Command3, Llama-44, DeepSeek-R15, Perplexity Sonar6, Mistral-Large-2.07) for accuracy and self-consistency across three runs. Results (i) Expert validation showed a consensus over MCQ transformation. For configuration 1, 98.1% (SD 0.01) of rephrased answers were appropriate, 98.1% (SD 0.01) of MCQ answers were correct, and 95.0% (SD 0.05) of rephrased questions were appropriate; corresponding values for configuration 2 were 96.3% (SD 0.05), 97.5% (SD 0.02), and 89.8% (SD 0.14). (ii) across benchmarking for configurations 1 and 2, most models achieved high accuracy (≥95%), but Mistral-Large-2.0, Llama 4, and Perplexity Sonar showed substantially lower self-consistency (60.4%-91.7%) than other models (≥95%). All models degraded in both accuracy and self-consistency on configuration 3. Conclusions DiaBench quantitatively differentiates LLM capability in T2D patient education, with Gemini, GPT-5, and DeepSeek-R1 performing strongly and other models suboptimally. DiaBench provides a transferable platform for evaluating existing and newly developed LLMs, addressing the need for standardized benchmarking before clinical deployment of LLM-based systems in patient education.