GPT-4 accurately classifies the clinical actionability of emergency department imaging reports

Abstract: Emergency department (ED) imaging volumes are increasing, necessitating prioritization of studies based on the probability of detecting actionable findings. We evaluated the performance of Generative Pretrained Transformer 4 (GPT-4), a large language model (LLM), in identifying clinically actionable findings in ED imaging reports. We analyzed 1,000 de-identified clinical impressions from the 20 most frequently ordered imaging studies at Stanford Health Care between 2020 and 2023. Two emergency physicians independently labeled each report for “clinically actionable findings” (likely to change ED management or disposition), with disagreements resolved by a third physician. GPT-4 labeled the same impressions using iterative prompt engineering. We calculated the accuracy, sensitivity, and specificity of GPT-4’s classifications compared to the adjudicated physician rating. The two physician raters agreed on actionability for 89.9% of studies. GPT-4 detected clinical actionability with 90.4% accuracy (84.0–98.0%, by study type), 88.8% sensitivity (75.0–100.0%), and 91.5% specificity (77.3–100.0%) compared to the adjudicated physician rating. GPT-4 accurately identified the clinical actionability of ED imaging studies, exhibiting slightly greater agreement with the adjudicated physician rating than the two physician raters did with one another. This approach allows labeling of large retrospective corpora, enabling development of data-driven tools to estimate pretest probability of actionable findings and prioritize imaging studies accordingly. LLMs can mimic physician gestalt in reducing high-dimensional data to a single meaningful measure, producing opportunities in clinical practice and research.