Development and Validation of a ChatGPT-Based Assessment for Conversation-Related Clinical Skills in Occupational Therapy OSCEs

ImportanceConversation-related clinical skills are central to Objective Structured Clinical Examinations (OSCEs) in occupational therapy. However, real-time examiner scoring is resource-intensive and may overlook students' specific skills. Transcript-based scoring with Chat Generative Pre-Trained Transformer (ChatGPT) may address these limitations.ObjectiveTo develop a ChatGPT-based assessment for chief-complaint interview OSCEs (G-ACE) and examine criterion validity against expert-consensus ratings, test-retest reliability, and inter-rater reliability.DesignSecondary analysis of de-identified transcripts from 8-min chief-complaint interviews with a simulated patient with stroke during an OSCE.SettingsA university in northern Taiwan.ParticipantsFifty-eight de-identified transcripts from undergraduate occupational therapy students.ResultsCriterion validity was acceptable for "Primary Symptoms and Related Feelings" and "Impact on Daily Life" items (weighted kappa ≥ 0.60), but lower for "Rehabilitation Expectations" and "Communication Skills" items (0.18-0.59). For domain and total scores, intraclass correlation coefficients (ICCs) were 0.17 to 0.78 and percentages of mean absolute errors (MAE%s) were 4.6% to 20.7%. Test-retest reliability was acceptable at the item level (weighted kappa = 0.68-1.00), and acceptable to good at the domain and total score levels (ICC = 0.65-0.85; MAE% = 2.3%-9.1%). Inter-rater reliability was acceptable for most domain and total scores (ICCs = 0.52-0.78; MAE%s = 1.3%-12.0%), except for the "Rehabilitation Expectations" domain (ICCs = 0.17-0.30; MAE%s = 1.3%-7.3%).Conclusion and RelevanceThe G-ACE demonstrated generally acceptable test-retest reliability and small absolute differences from expert ratings for domain and total scores. Because some items showed lower consistency with expert ratings, formative feedback should be interpreted cautiously based on transcript evidence.