Comparative performance of Chinese and international large language models in patient education on testicular torsion: a cross-sectional study

Background Patients and caregivers increasingly rely on LLMs for health information, yet their performance in TT, a time-critical urological emergency, remains insufficiently characterized. This study compared three LLMs across content safety, quality, reliability, readability, understandability, and actionability. Methods Twenty-five standardized patient-centered questions were developed based on clinical guidelines, expert consensus documents, and information needs identified through Google Trends. Each question was submitted to GPT-5.6 Sol (High reasoning mode), DeepSeek-V4-Pro, and Doubao 2.0 Expert Model, yielding 75 responses. Quality and reliability were assessed using GQS, mDISCERN, and QAMAI; understandability and actionability were evaluated using PEMAT-U and PEMAT-A, respectively. Readability was assessed using six English-language indices, and content deficiencies were also evaluated. Results GPT-5.6 Sol(High reasoning mode) achieved significantly higher GQS, mDISCERN, and QAMAI scores than the other two models (all overall P < 0.001). Hallucinations and factual inaccuracies occurred in 28.0% and 17.3% of responses, respectively. No hallucinations were detected in GPT-5.6 Sol responses, compared with 44.0% for DeepSeek-V4-Pro and 40.0% for Doubao 2.0 Expert Model. Understandability did not differ significantly among models ( P = 0.0735), whereas actionability did ( P < 0.001), with GPT-5.6 Sol(High reasoning mode) performing best. All six readability indices differed significantly among models (all P < 0.001). DeepSeek-V4-Pro showed the most favorable relative readability, GPT-5.6 Sol(High reasoning mode) intermediate performance, and Doubao 2.0 Expert Model the greatest reading difficulty. Conclusions The three LLMs exhibited distinct performance profiles in generating TT-related health information. GPT-5.6 Sol (High reasoning mode) achieved the highest scores for quality, reliability, and actionability, whereas DeepSeek-V4-Pro showed comparatively better readability. Understandability did not differ significantly among models. However, hallucinations, factual inaccuracies, and substantial reading demands remain concerns. These findings reflect model performance under the specific evaluation conditions and at the time of data collection, rather than established clinical effectiveness or safety. GPT-5.6 Sol(High reasoning mode) may be the most suitable supplementary information tool among the models evaluated, but it should not replace professional medical assessment or clinical decision-making.

Authors

Institutions

Publication Details

Journal
Frontiers in Medicine
Published
2026-11-18
DOI
https://doi.org/10.3389/fmed.2026.1978486
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Comparative performance of Chinese and international large language models in patient education on testicular torsion: a cross-sectional study

Fazhong Dai, Bingwei Wang
3 links
Frontiers in Medicine
Artificial Intelligence in Healthcare and Education
article

Comparative performance of Chinese and international large language models in patient education on testicular torsion: a cross-sectional study

Fazhong Dai, Bingwei Wang
article en

Abstract

Background Patients and caregivers increasingly rely on LLMs for health information, yet their performance in TT, a time-critical urological emergency, remains insufficiently characterized. This study compared three LLMs across content safety, quality, reliability, readability, understandability, and actionability. Methods Twenty-five standardized patient-centered questions were developed based on clinical guidelines, expert consensus documents, and information needs identified through Google Trends. Each question was submitted to GPT-5.6 Sol (High reasoning mode), DeepSeek-V4-Pro, and Doubao 2.0 Expert Model, yielding 75 responses. Quality and reliability were assessed using GQS, mDISCERN, and QAMAI; understandability and actionability were evaluated using PEMAT-U and PEMAT-A, respectively. Readability was assessed using six English-language indices, and content deficiencies were also evaluated. Results GPT-5.6 Sol(High reasoning mode) achieved significantly higher GQS, mDISCERN, and QAMAI scores than the other two models (all overall P < 0.001). Hallucinations and factual inaccuracies occurred in 28.0% and 17.3% of responses, respectively. No hallucinations were detected in GPT-5.6 Sol responses, compared with 44.0% for DeepSeek-V4-Pro and 40.0% for Doubao 2.0 Expert Model. Understandability did not differ significantly among models ( P = 0.0735), whereas actionability did ( P < 0.001), with GPT-5.6 Sol(High reasoning mode) performing best. All six readability indices differed significantly among models (all P < 0.001). DeepSeek-V4-Pro showed the most favorable relative readability, GPT-5.6 Sol(High reasoning mode) intermediate performance, and Doubao 2.0 Expert Model the greatest reading difficulty. Conclusions The three LLMs exhibited distinct performance profiles in generating TT-related health information. GPT-5.6 Sol (High reasoning mode) achieved the highest scores for quality, reliability, and actionability, whereas DeepSeek-V4-Pro showed comparatively better readability. Understandability did not differ significantly among models. However, hallucinations, factual inaccuracies, and substantial reading demands remain concerns. These findings reflect model performance under the specific evaluation conditions and at the time of data collection, rather than established clinical effectiveness or safety. GPT-5.6 Sol(High reasoning mode) may be the most suitable supplementary information tool among the models evaluated, but it should not replace professional medical assessment or clinical decision-making.

Frontiers in MedicineVol. 13
Third Affiliated Hospital of Guangzhou Medical University (CN), Guangzhou Medical University (CN)
Good health and well-being
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon