Guideline Concordance, Information Quality, and Readability of Large Language Model Responses to Female Urethral Stricture Questions: A Comparative Cross-sectional Study

Aim Large language models (LLMs) are increasingly employed to acquire medical information. However, their reliability in addressing rare and technically intricate conditions, such as female urethral stricture disease (FUSD), remains uncertain. This study evaluated the guideline concordance, information quality, reliability, and readability of responses generated by three widely accessible LLMs in response to clinical questions related to FUSD. Methods This comparative cross-sectional study was conducted in January 2026. Thirty standardized questions across five clinical domains were developed based on the 2025 European Association of Urology Guidelines for Urethral Stricture Disease and refined through expert consensus. These questions were submitted to ChatGPT 5.0, Gemini 3.0, and Microsoft Copilot (Basic) under identical conditions. Responses were independently evaluated by three blinded board-certified urologists using a 5-point expert-rated accuracy scale. Information reliability and quality were assessed using modified DISCERN and ensuring quality information for patients (EQIP) independently by the blinded reviewers, while readability was evaluated using flesch reading ease score (FRES) and flesch-kincaid grade level (FKGL). Between-model comparisons were performed using the Friedman test, followed by Wilcoxon signed-rank tests with Bonferroni correction. Results Gemini achieved the highest expert-rated accuracy (4.80±0.41) and modified DISCERN score (19.0±1.7) among the three models. The Copilot achieved the highest EQIP (89.60±4.01) and FRES (42.23±3.89) scores and the lowest FKGL (12.67±1.04). Significant differences between the models were identified across all evaluated metrics. Gemini had significantly higher modified DISCERN scores than ChatGPT and Copilot (both p<0.001), whereas ChatGPT and Copilot did not differ significantly. Conclusion The evaluated LLMs demonstrated distinct performance profiles rather than uniform superiority. Gemini showed the highest expert-rated accuracy and information reliability, whereas Copilot provided the most favorable patient-oriented information quality and readability. Large language model-generated information on FUSD should be verified against current clinical guidance before clinical or patient education.

Authors

Institutions

Publication Details

Journal
Medical Bulletin of Haseki
Published
2026-11-17
DOI
https://doi.org/10.4274/haseki.galenos.2026.93585
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Guideline Concordance, Information Quality, and Readability of Large Language Model Responses to Female Urethral Stricture Questions: A Comparative Cross-sectional Study

Halil Lütfi Canat, Metin Savun, Yiğit Can Filtekin, Çağrı Şevik et al.
5 links
Medical Bulletin of Haseki
Artificial Intelligence in Healthcare and Education
article

Guideline Concordance, Information Quality, and Readability of Large Language Model Responses to Female Urethral Stricture Questions: A Comparative Cross-sectional Study

Halil Lütfi Canat, Metin Savun, Yiğit Can Filtekin, Çağrı Şevik, Süleyman Şahin
article en

Abstract

Aim Large language models (LLMs) are increasingly employed to acquire medical information. However, their reliability in addressing rare and technically intricate conditions, such as female urethral stricture disease (FUSD), remains uncertain. This study evaluated the guideline concordance, information quality, reliability, and readability of responses generated by three widely accessible LLMs in response to clinical questions related to FUSD. Methods This comparative cross-sectional study was conducted in January 2026. Thirty standardized questions across five clinical domains were developed based on the 2025 European Association of Urology Guidelines for Urethral Stricture Disease and refined through expert consensus. These questions were submitted to ChatGPT 5.0, Gemini 3.0, and Microsoft Copilot (Basic) under identical conditions. Responses were independently evaluated by three blinded board-certified urologists using a 5-point expert-rated accuracy scale. Information reliability and quality were assessed using modified DISCERN and ensuring quality information for patients (EQIP) independently by the blinded reviewers, while readability was evaluated using flesch reading ease score (FRES) and flesch-kincaid grade level (FKGL). Between-model comparisons were performed using the Friedman test, followed by Wilcoxon signed-rank tests with Bonferroni correction. Results Gemini achieved the highest expert-rated accuracy (4.80±0.41) and modified DISCERN score (19.0±1.7) among the three models. The Copilot achieved the highest EQIP (89.60±4.01) and FRES (42.23±3.89) scores and the lowest FKGL (12.67±1.04). Significant differences between the models were identified across all evaluated metrics. Gemini had significantly higher modified DISCERN scores than ChatGPT and Copilot (both p<0.001), whereas ChatGPT and Copilot did not differ significantly. Conclusion The evaluated LLMs demonstrated distinct performance profiles rather than uniform superiority. Gemini showed the highest expert-rated accuracy and information reliability, whereas Copilot provided the most favorable patient-oriented information quality and readability. Large language model-generated information on FUSD should be verified against current clinical guidance before clinical or patient education.

Medical Bulletin of Haseki
Sağlık Bilimleri Üniversitesi (TR), İstanbul Başakşehir Çam ve Sakura Şehir Hastanesi (TR), University of Health Sciences Antigua (AG)
Good health and well-being
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon