Hierarchical Prompting with Reasoning Large Language Models for Immune Checkpoint Inhibitor-associated Myocarditis Evidence Extraction Pilot Study

Abstract Background Immunotherapy with immune checkpoint inhibitors (ICIs) is an effective treatment for many cancers, but it can induce immune-related adverse events (irAEs). Among those, myocarditis is a severe complication associated with high morbidity and mortality. To support the rapidly emerging research, labor-intense manual data extraction is often needed. Recent studies adopted large language models (LLMs) to systematically process clinical notes and identify patients with irAEs. However, research gaps persist: 1) For ICI-associated myocarditis, the applicability and scalability of LLMs have not been systematically evaluated; 2) most existing studies focused on patient-level irAEs detection while omitting the clinical nuances; 3) cutting-edge LLM techniques have not been adopted. Objectives Evaluate an LLM approach to extract ICI-associated myocarditis evidence, including inflammatory infiltrate, life-threatening arrhythmias, heart failure, and stroke from clinical notes. Methods We collected clinical notes of patients with ICI-associated myocarditis at a single center to develop and evaluate LLM-based methods that convert free-text clinical notes into structured data with entities (e.g., diagnosis, treatment, and imaging) and attributes (e.g., date, assertion, and status). We systematically evaluated three techniques: LLM reasoning, context engineering, and hierarchical prompting against ground truth created by a cardiooncologist. Results Our proposed sentence-based context engineering and hierarchical prompting method, with a reasoning LLM, is significantly more accurate than the research trainees (F1 score of 0.6332 vs. 0.4933, p<0.001), faster (37 minutes vs. 72 hours per 1000 notes), and more cost-effective ($2.04 USD per 1000 notes). Conclusions We proposed an optimized method that integrates hierarchical prompting, context engineering, and a reasoning LLM for ICI-associated myocarditis evidence extraction to support physicians and researchers in rapid clinical data collection for research to make diagnostic and treatment inferences.