Artificial Intelligence‐Driven Extraction of Patient‐Reported Pain Outcomes in Osteoarthritis Using Large Language Models

OBJECTIVE: Patient-reported outcome measures (PROMs) are central to assessing symptom burden and treatment response in Osteoarthritis (OA) but are often embedded in unstructured clinical notes. We evaluated whether prompt-engineered large language models (LLMs) could accurately extract or infer pain scores on a 0-10 scale from routine OA clinical documentation. METHODS: We retrospectively analyzed 163 paired internal medicine and orthopedic clinic notes from patients in a prospective OA registry. A trained abstractor extracted explicitly documented pain scores from notes. For notes lacking explicit pain scores, physicians estimated pain severity from narrative documentation. The Llama 3.2-90B model, using Chain-of-Thought and Few-Shot prompting, inferred pain scores. Registry-collected pain scores served as a reference standard. Agreement was assessed using Kendall's Tau-b. RESULTS: Agreement between LLM-extracted and abstracted scores was strong for IM notes (n =132, τ = 0.76, p < 0.0001) and excellent for orthopedic notes (n = 65, τ = 0.95, p < 0.0001). The LLM identified 15 EMR-generated default zero scores in IM notes; after excluding them, agreement was perfect (τ = 1.0, p < 0.0001). For notes without explicit pain scores (31 IM and 98 orthopedic notes), LLM-based pain inference showed weak correlation with registry scores (IM: τ = 0.17; orthopedic: τ = 0.11), but performed comparably to physician estimation. LLM-physician agreement ranged from 54.2%-88.0% within ±1 point and 66.7%-100% within ±2 points for IM notes, and from 54.8%-70.5% within ±1 point and 84.5%-97.7% within ±2 points for orthopedic notes. CONCLUSION: LLMs accurately extract explicitly documented PROMs from clinical notes and reliably identify documentation artifacts. Inference from narrative documentation was less precise when notes lacked sufficient contextual information and detailed pain descriptions.