Feasibility of Using Clinicians to Code the Actionability of Narrative Comments About Patient Experience for Quality Improvement

BACKGROUND AND OBJECTIVES: Many healthcare organizations administer Consumer Assessment of Healthcare Providers and Systems (CAHPS) surveys and use both quantitative ratings and qualitative narrative comments. While narrative data offer rich insights into patient experience, their time-intensive analysis often limits systematic use in quality improvement (QI). Healthcare organizations often underutilize this data, reading comments and not analyzing them systematically. This study examined the feasibility and accuracy of engaging clinicians in systematically coding narrative CAHPS data for QI. We conducted a coding exercise using patient-level narrative comments and compared the inter-rater reliability of clinicians and experienced qualitative researchers. METHODS: We partnered with a large, urban freestanding children's hospital that routinely collects Child HCAHPS data. Our six-person coding team comprised two experienced qualitative researchers, two pediatric inpatient physicians, and two pediatric inpatient nurses. Following standardized training on the coding process and a predefined codebook, the comments were coded independently. The coding framework included valence (positive, negative, mixed), identification of potential patient safety issues (safety flag), staff type mentioned (e.g., doctor, nurse), setting mentioned (e.g., emergency room), overall actionability (yes/no), and, if actionable, the level of action needed (individual clinician/provider or organizational). We calculated overall and pairwise inter-rater agreement (Kappa) to assess coding consistency across and between the three professional groups and for different code combinations to identify patterns. RESULTS: Researchers had excellent inter-rater agreement (pooled Kappa = 0.88). There was very good agreement among clinicians (Kappa = 0.68), whereas nurses had good agreement (Kappa = 0.60). Valence and mentioned staff type were coded with high consistency across all groups. Agreement on the dichotomous 0/1 indicator of actionability (yes/no) was higher than for the three-level coding of actionability (not actionable/actionable at provider-level/actionable at organizational-level). Notably, nurses displayed slightly higher agreement than physicians, whereas researchers were consistently more concordant than both clinician groups. Nurses also demonstrated the strongest sensitivity in identifying potential patient safety issues within the narrative comments. CONCLUSION: Our findings suggest that training nurses to code narrative CAHPS comments for valence, mentioned staff, overall actionability, and patient safety flags is a feasible and potentially effective strategy for QI. This focused coding approach, implemented regularly (e.g., quarterly), could efficiently generate a substantial dataset for integration into routine QI initiatives and staff meetings.