Use of natural language processing tools in musculoskeletal disability assessment: generating reports and calculating impairment percentages in Turkish health commission settings

PURPOSE: This study explores the potential of Natural Language Processing (NLP) tools, specifically ChatGPT-4o and Data Analyst, in supporting health commissions with disability assessments. METHODS: Nine realistic patient scenarios were created to reflect typical cases in disability evaluations, encompassing conditions like stroke, cerebral palsy, traumatic injuries, and hereditary disorders, each involving varied motor and functional impairments. These scenarios were input into ChatGPT-4o and Data Analyst. Their outputs were evaluated using a 5-point Likert scale on alignment with expert guidelines, completeness, and lack of false information. Interrater reliability was established before scoring. RESULTS: Both models produced generally high-quality narrative reports, ChatGPT-4o was rated "very good" and Data Analyst "good," with no statistically significant difference in overall scores. However, ChatGPT-4o failed to calculate correct disability percentages in 55.6%, and Data Analyst failed in 88.9% of scenarios. CONCLUSIONS: While NLP tools can assist in generating structured disability reports, they currently lack the precision needed for calculating disability percentages reliably. Expert oversight remains essential for decision-making in disability assessments.