A dataset and benchmark for hospital course summarization with adapted large language models
We release a foundational clinically relevant dataset, the MIMIC-IV-BHC, and present an open-source benchmark of LLM performance in BHC synthesis from clinical notes. We observe high-quality summarization performance for both in-context proprietary and fine-tuned open-source LLMs using both quantitative metrics and a qualitative clinical reader study. Our research effectively integrates elements from the data assimilation pipeline: our methods use (1) clinical data sources to integrate, (2) data translation, and (3) knowledge creation, while our evaluation strategy paves the way for (4) deployment.
Authors
- N. Clifford (ORCID: https://orcid.org/0000-0003-2334-3818)
- Jangwon Kim (ORCID: https://orcid.org/0000-0003-0228-3502)
- Eduardo Pontes Reis (ORCID: https://orcid.org/0000-0001-5110-457X)
- Asad Aali (ORCID: https://orcid.org/0009-0008-2120-5722)
- Joseph Daws
- Jason Hom (ORCID: https://orcid.org/0000-0002-8404-5796)
- Akshay Sanjay Chaudhari (ORCID: https://orcid.org/0000-0002-3667-6796)
- Yamin Arefeen (ORCID: https://orcid.org/0000-0003-2194-1945)
- Dave Van Veen (ORCID: https://orcid.org/0000-0001-9312-1773)
- Sergios Gatidis (ORCID: https://orcid.org/0000-0002-6928-4967)
- Christian Blüthgen (ORCID: https://orcid.org/0000-0001-7321-5676)
- Arash Saber Tehrani
Institutions
- Amazon (United States) (US)
- Hospital Israelita Albert Einstein (BR)
- Association for the Advancement of Artificial Intelligence (US)
- University Hospital of Zurich (CH)
- University of California San Francisco Medical Center (US)
- Hospital São Paulo (BR)
- Artificial Intelligence in Medicine (Canada) (CA)
- The University of Texas at Austin (US)
- Stanford University (US)
Publication Details
- Journal
- Journal of the American Medical Informatics Association
- Published
- 2024-12-30
- DOI
- https://doi.org/10.1093/jamia/ocae312
- Citations
- 22
- Primary Topic
- Topic Modeling
- Type
- article
- Field-Weighted Citation Impact
- 7.64
Funders
- National Institutes of Health
- Stanford Cardiovascular Institute, School of Medicine, Stanford University