Orthopedic hand surgery: Evaluating the ability of chat generative pretrained transformer to improve grade-level readability in patient education materials
Background: The American Medical Association recommends patient education materials be written at a sixth-grade reading level. We hypothesized that a chat generative pretrained transformer (ChatGPT) could simplify educational materials to better align with this guideline. Methods: All 95 patient education handouts from the American Society for Surgery of the Hand were evaluated using validated readability tools: Flesch-Kincaid (FK), Automated Readability Index, Simple Measure of Gobbledygook, Gunning Fog Index, and Coleman-Liau Index, and a validated understandability and actionability assessment: Patient Education Material Assessment Tool. Handouts were processed through OpenAI’s ChatGPT 3.5, 4, and 4o with the prompt to translate to a sixth-grade reading level. Revised outputs were reassessed with the same metrics and evaluated for content preservation. Results: ChatGPT improved composite grade-level readability (8.6 vs. 10.5) but remained above the sixth-grade target. ChatGPT 4o improved readability in nearly all metrics: FK grade level (8.3 vs. 10.3), FK reading ease (68.0 vs. 54.9), Automated Readability Index (7.4 vs. 9.4), Simple Measure of Gobbledygook (8.3 vs. 10.2), Coleman-Liau Index(9.3 vs. 10.9), and Gunning Fog Index (9.6 vs. 11.9). However, ChatGPT-generated materials received lower total Patient Education Material Assessment Tool scores (71.1% vs. 79.6%). Changes in readability were primarily due to lower total word count (410 vs. 719), shorter sentence length (12 vs. 21 words per sentence), and shorter word length (147 vs. 155 syllables per 100 words). ChatGPT preserved content of the original handouts with a maximum rate of 94.7%. Conclusions: ChatGPT increased readability while preserving content. However, it did not reach the target sixth-grade reading level.
Authors
- Julia A.V. Nuelle (ORCID: https://orcid.org/0000-0001-7921-8771)
- Daniel A. London (ORCID: https://orcid.org/0000-0002-1711-0762)
- Benjamin J. Kirby (ORCID: https://orcid.org/0000-0003-4060-2471)
- Peter Serour
- Samuel Nofsinger
- Peter A. Albrecht
Institutions
- University of Missouri (US)
Publication Details
- Journal
- Current Orthopaedic Practice
- Published
- 2026-11-18
- DOI
- https://doi.org/10.1097/bco.0000000000001407
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 0.00