A comparative analysis of large language models versus traditional information extraction methods for real-world evidence of patient symptomatology in acute and post-acute sequelae of SARS-CoV-2
While BioMedICUS generally outperformed the LLMs, the latter showed promising results in specific areas, particularly LLaMA3-8B, in identifying negative symptom mentions. However, both LLaMA models faced challenges in demographic fairness and generalizability. These findings underscore the need for diverse, high-quality training datasets and robust annotation processes to enhance LLMs' performance and reliability in clinical applications.
Authors
- Genevieve B. Melton (ORCID: https://orcid.org/0000-0001-5193-1663)
- Nicholas E. Ingraham (ORCID: https://orcid.org/0000-0002-0292-0594)
- Greg Silverman (ORCID: https://orcid.org/0000-0003-4844-1473)
- Emma Jones (ORCID: https://orcid.org/0000-0003-4083-0220)
- Vedansh Thakkar (ORCID: https://orcid.org/0009-0001-3551-5229)
- Christopher J. Tignanelli
- Abhinab Kc
- Rui Zhang (ORCID: https://orcid.org/0000-0001-8258-3585)
- Samantha King (ORCID: https://orcid.org/0000-0001-9138-1838)
Institutions
- University of Minnesota (US)
- University of Washington (US)
- University of Minnesota System (US)
- University of Minnesota Medical Center (US)
- Minnesota Department of Health (US)
Publication Details
- Journal
- PLoS ONE
- Published
- 2025-05-15
- DOI
- https://doi.org/10.1371/journal.pone.0323535
- Citations
- 2
- Primary Topic
- Machine Learning in Healthcare
- Type
- article
- Field-Weighted Citation Impact
- 4.96
Funders
- National Center for Complementary and Integrative Health