Orthopedic hand surgery: Evaluating the ability of chat generative pretrained transformer to improve grade-level readability in patient education materials

Background: The American Medical Association recommends patient education materials be written at a sixth-grade reading level. We hypothesized that a chat generative pretrained transformer (ChatGPT) could simplify educational materials to better align with this guideline. Methods: All 95 patient education handouts from the American Society for Surgery of the Hand were evaluated using validated readability tools: Flesch-Kincaid (FK), Automated Readability Index, Simple Measure of Gobbledygook, Gunning Fog Index, and Coleman-Liau Index, and a validated understandability and actionability assessment: Patient Education Material Assessment Tool. Handouts were processed through OpenAI’s ChatGPT 3.5, 4, and 4o with the prompt to translate to a sixth-grade reading level. Revised outputs were reassessed with the same metrics and evaluated for content preservation. Results: ChatGPT improved composite grade-level readability (8.6 vs. 10.5) but remained above the sixth-grade target. ChatGPT 4o improved readability in nearly all metrics: FK grade level (8.3 vs. 10.3), FK reading ease (68.0 vs. 54.9), Automated Readability Index (7.4 vs. 9.4), Simple Measure of Gobbledygook (8.3 vs. 10.2), Coleman-Liau Index(9.3 vs. 10.9), and Gunning Fog Index (9.6 vs. 11.9). However, ChatGPT-generated materials received lower total Patient Education Material Assessment Tool scores (71.1% vs. 79.6%). Changes in readability were primarily due to lower total word count (410 vs. 719), shorter sentence length (12 vs. 21 words per sentence), and shorter word length (147 vs. 155 syllables per 100 words). ChatGPT preserved content of the original handouts with a maximum rate of 94.7%. Conclusions: ChatGPT increased readability while preserving content. However, it did not reach the target sixth-grade reading level.

Authors

Institutions

Publication Details

Journal
Current Orthopaedic Practice
Published
2026-11-18
DOI
https://doi.org/10.1097/bco.0000000000001407
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Orthopedic hand surgery: Evaluating the ability of chat generative pretrained transformer to improve grade-level readability in patient education materials

Julia A.V. Nuelle, Daniel A. London, Benjamin J. Kirby, Peter Serour et al.
6 links
Current Orthopaedic Practice
Artificial Intelligence in Healthcare and Education
article

Orthopedic hand surgery: Evaluating the ability of chat generative pretrained transformer to improve grade-level readability in patient education materials

Julia A.V. Nuelle, Daniel A. London, Benjamin J. Kirby, Peter Serour, Samuel Nofsinger, Peter A. Albrecht
article en

Abstract

Background: The American Medical Association recommends patient education materials be written at a sixth-grade reading level. We hypothesized that a chat generative pretrained transformer (ChatGPT) could simplify educational materials to better align with this guideline. Methods: All 95 patient education handouts from the American Society for Surgery of the Hand were evaluated using validated readability tools: Flesch-Kincaid (FK), Automated Readability Index, Simple Measure of Gobbledygook, Gunning Fog Index, and Coleman-Liau Index, and a validated understandability and actionability assessment: Patient Education Material Assessment Tool. Handouts were processed through OpenAI’s ChatGPT 3.5, 4, and 4o with the prompt to translate to a sixth-grade reading level. Revised outputs were reassessed with the same metrics and evaluated for content preservation. Results: ChatGPT improved composite grade-level readability (8.6 vs. 10.5) but remained above the sixth-grade target. ChatGPT 4o improved readability in nearly all metrics: FK grade level (8.3 vs. 10.3), FK reading ease (68.0 vs. 54.9), Automated Readability Index (7.4 vs. 9.4), Simple Measure of Gobbledygook (8.3 vs. 10.2), Coleman-Liau Index(9.3 vs. 10.9), and Gunning Fog Index (9.6 vs. 11.9). However, ChatGPT-generated materials received lower total Patient Education Material Assessment Tool scores (71.1% vs. 79.6%). Changes in readability were primarily due to lower total word count (410 vs. 719), shorter sentence length (12 vs. 21 words per sentence), and shorter word length (147 vs. 155 syllables per 100 words). ChatGPT preserved content of the original handouts with a maximum rate of 94.7%. Conclusions: ChatGPT increased readability while preserving content. However, it did not reach the target sixth-grade reading level.

Current Orthopaedic Practice
University of Missouri (US)
Good health and well-being, Quality education
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon