Reliability and Readability of AI Chatbot Responses on Venous Thromboembolism After Fracture Surgery: A Comparative Study

Abstract Background Venous thromboembolism (VTE) is an important complication after fracture surgery, and patients increasingly seek health information through internet based resources and artificial intelligence (AI) chatbots. However, the reliability and readability of AI generated information about postoperative VTE remain unclear. This study evaluated public information needs regarding VTE and assessed the reliability and readability of information provided by four widely used AI chatbots. Methods Google Trends was used to characterize worldwide public search interest in VTE from August 2021 to August 2026. Twenty patient oriented questions were developed based on popular searches and clinical experience and submitted verbatim to ChatGPT-5.6 Sol, Claude Sonnet 5, DeepSeek-V4-Pro, and Gemini 3.6 Flash. Reliability was assessed using DISCERN, EQIP, GQS, and JAMA criteria, while readability was evaluated using six established indices. Results Global search interest in VTE increased markedly from mid 2025 and peaked in July 2026. Reliability differed significantly among the four chatbots across all four assessment tools, with ChatGPT showing the highest median DISCERN, EQIP, and JAMA scores; all omnibus comparisons remained significant after Holm–Šídák correction (adjusted P < 0.0001). Readability remained poor across all models, with grade level indices well above the recommended sixth grade benchmark and FRES values ranging from 28.00 to 36.35, far below the recommended threshold of 80. Conclusion Public interest in VTE has increased in recent years, and contemporary AI chatbots differ in the reliability of information provided about VTE after fracture surgery. ChatGPT showed higher reliability scores across several domains, whereas source transparency remained limited across models. Responses from all four chatbots were substantially more complex than the recommended reading level for patient education. Improvements in reliability, source transparency, and readability are needed to enhance the usefulness of AI generated information for patient education.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-12
DOI
https://doi.org/10.21203/rs.3.rs-11424655/v1
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
preprint

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Reliability and Readability of AI Chatbot Responses on Venous Thromboembolism After Fracture Surgery: A Comparative Study

Mengyue Xie, Li Yi, Zhang Chuanliang, Xiao Ming et al.
Research Square
Artificial Intelligence in Healthcare and Education
preprint

Reliability and Readability of AI Chatbot Responses on Venous Thromboembolism After Fracture Surgery: A Comparative Study

Mengyue Xie, Li Yi, Zhang Chuanliang, Xiao Ming, Xiaohua Wu, Yingsi Zhang
preprint en

Abstract

Abstract Background Venous thromboembolism (VTE) is an important complication after fracture surgery, and patients increasingly seek health information through internet based resources and artificial intelligence (AI) chatbots. However, the reliability and readability of AI generated information about postoperative VTE remain unclear. This study evaluated public information needs regarding VTE and assessed the reliability and readability of information provided by four widely used AI chatbots. Methods Google Trends was used to characterize worldwide public search interest in VTE from August 2021 to August 2026. Twenty patient oriented questions were developed based on popular searches and clinical experience and submitted verbatim to ChatGPT-5.6 Sol, Claude Sonnet 5, DeepSeek-V4-Pro, and Gemini 3.6 Flash. Reliability was assessed using DISCERN, EQIP, GQS, and JAMA criteria, while readability was evaluated using six established indices. Results Global search interest in VTE increased markedly from mid 2025 and peaked in July 2026. Reliability differed significantly among the four chatbots across all four assessment tools, with ChatGPT showing the highest median DISCERN, EQIP, and JAMA scores; all omnibus comparisons remained significant after Holm–Šídák correction (adjusted P < 0.0001). Readability remained poor across all models, with grade level indices well above the recommended sixth grade benchmark and FRES values ranging from 28.00 to 36.35, far below the recommended threshold of 80. Conclusion Public interest in VTE has increased in recent years, and contemporary AI chatbots differ in the reliability of information provided about VTE after fracture surgery. ChatGPT showed higher reliability scores across several domains, whereas source transparency remained limited across models. Responses from all four chatbots were substantially more complex than the recommended reading level for patient education. Improvements in reliability, source transparency, and readability are needed to enhance the usefulness of AI generated information for patient education.

Research Square
Peking University Shenzhen Hospital (CN)
Peking University, National Health Commission of the People's Republic of China
Good health and well-being
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon