Uncertainty estimation in diagnosis generation from large language models: next-word probability is not pre-test probability
LLMs demonstrate potential but currently fall short in estimating diagnostic probabilities compared to traditional machine learning classifiers trained on structured EHR data. Further improvements are needed for reliable clinical use.
Authors
- Anoop Mayampurath (ORCID: https://orcid.org/0000-0002-3010-6960)
- Skatje Myers (ORCID: https://orcid.org/0000-0001-9524-8436)
- Danielle S. Bitterman (ORCID: https://orcid.org/0000-0003-0345-2232)
- Timothy A. Miller (ORCID: https://orcid.org/0000-0003-4513-403X)
- Dmitriy Dligach (ORCID: https://orcid.org/0000-0002-2585-2707)
- Majid Afshar (ORCID: https://orcid.org/0000-0002-6368-4652)
- Shan Chen (ORCID: https://orcid.org/0000-0001-7999-7410)
- Matthew M. Churpek (ORCID: https://orcid.org/0000-0002-4030-5250)
- Guanhua Chen (ORCID: https://orcid.org/0000-0002-9314-2037)
- Yanjun Gao (ORCID: https://orcid.org/0000-0002-9341-7360)
Institutions
- Brigham and Women's Hospital (US)
- Boston Children's Hospital (US)
- University of Wisconsin–Madison (US)
- Harvard University (US)
- Loyola University Chicago (US)
- Dana-Farber Cancer Institute (US)
- Dana-Farber Brigham Cancer Center (US)
- Mass General Brigham (US)
- University of Colorado Anschutz Medical Campus (US)
Publication Details
- Journal
- JAMIA Open
- Published
- 2024-12-26
- DOI
- https://doi.org/10.1093/jamiaopen/ooae154
- Citations
- 6
- Primary Topic
- Clinical Reasoning and Diagnostic Skills
- Type
- article
- Field-Weighted Citation Impact
- 3.02
Funders
- National Science Foundation
- National Institutes of Health
- School of Medicine and Public Health, University of Wisconsin-Madison
- National Heart, Lung, and Blood Institute
- U.S. National Library of Medicine