Human-in-the-loop large language model framework for identification of cutaneous immune-related adverse events
This study evaluated a retrieval-augmented, multi-agent large language model (LLM)–driven, human-in-the-loop framework for detecting cutaneous immune-related adverse events (cirAEs) from clinical notes. Compared with the unassisted manual-review group, the LLM-assisted group demonstrated higher accuracy (F1 = 0.88 vs 0.77) and inter-rater agreement measured by Cohen’s κ ( κ = 0.82 vs 0.50), and reported an approximately one-third shorter average review time. This framework pilots how LLMs can be applied to identify immune-related toxicities and, more broadly, support accurate, scalable, and transparent adverse event data extraction.
Authors
- Eudora Lee
- Nicole LeBoeuf
- C. Duffy
- Alexander Gusev (ORCID: https://orcid.org/0000-0002-7980-4620)
- Lirit Fuksman
- Kun‐Hsing Yu (ORCID: https://orcid.org/0000-0001-9892-8218)
- Zeyun Lu (ORCID: https://orcid.org/0000-0002-3511-6850)
- Ralina Karagenova
- Olivia Burke
- Adam Kashlan
- Priya Katyal
- Andrew Sedlack
- Debby Cheng
- Jin Ning Tian
- Yevgeniy R. Semenov
- Charles Lu
- Chuck Lin
Institutions
- Brigham and Women's Hospital (US)
- The University of Texas MD Anderson Cancer Center (US)
- Harvard University (US)
- Massachusetts General Hospital (US)
- Dana-Farber Cancer Institute (US)
- Dana-Farber/Harvard Cancer Center (US)
Publication Details
- Journal
- npj Precision Oncology
- Published
- 2026-10-15
- DOI
- https://doi.org/10.1038/s41698-026-01737-y
- Primary Topic
- Pharmacovigilance and Adverse Drug Reactions
- Type
- article
- Field-Weighted Citation Impact
- 0.00
Funders
- U.S. Department of Defense
- Melanoma Research Foundation
- Melanoma Research Alliance
- National Heart, Lung, and Blood Institute
- National Institute of General Medical Sciences
- National Institute of Arthritis and Musculoskeletal and Skin Diseases