NCOG-03. Scalable Tracking of Symptoms in the Electronic Health Record using Large Language Models in Patients with Central Nervous System Cancers Undergoing Therapy

Abstract BACKGROUND Advances in large language models (LLMs) provide a means for scalable tracking of patient symptoms in clinical trials and post-marking surveillance using the electronic health record (EHR). We sought to validate symptoms extracted from the EHR using a LLM to scale symptom extraction from the EHR. METHODS Across a dataset of 499 randomly chosen clinical notes from patients seen in a neuro-oncology clinic, physician annotations were compared to LLM annotations (GPT-4o) using nine annotated symptoms (headache, fatigue, nausea, anxiety, difficulties sleeping, numbness and tingling, rash, constipation, and diarrhea). We applied the LLM to an external dataset of 51,541 notes representing 1,642 patients seen in neuro-oncology clinic (1/2018 to 12/2023) to obtain real-world symptom prevalence for temozolomide, bevacizumab, lomustine, immune checkpoint inhibitors (ICI), and methotrexate. RESULTS Across all symptoms, the LLM performance allowed us to generalize the model to the larger external dataset (F1-score: 0.97, n=499 notes). In the external dataset, the average age was 57.1 years old. Each patient was presented by 13.5 notes, on average. Psychologist reported the highest number of any symptoms (74.1%, 1+ symptoms), and physicians reported the highest number of total symptoms (7.5%, 4+ symptoms). The average number of symptoms per note was 3.92, and the most common symptom was fatigue (83% of patients). Surprisingly, patients receiving ICIs suffered from the most symptoms (mean = 4.68) and those receiving methotrexate had the least (mean = 2.92). We also found that the prevalence of reported symptoms in this real-world cohort was often much greater than the prevalence of reported symptoms in clinical trials of similar treatment regimens (range 4 to 250 times greater). CONCLUSION In conclusion, LLMs offer the ability to scale symptom extraction from health records, which is crucial to understand symptom burden and power symptom-related interventions and studies in real-world patient cohorts.