Privacy-preserving large language models for structured medical information retrieval
Most clinical information is encoded as free text, not accessible for quantitative analysis. This study presents an open-source pipeline using the local large language model (LLM) "Llama 2" to extract quantitative information from clinical text and evaluates its performance in identifying features of decompensated liver cirrhosis. The LLM identified five key clinical features in a zero- and one-shot manner from 500 patient medical histories in the MIMIC IV dataset. We compared LLMs of three sizes and various prompt engineering approaches, with predictions compared against ground truth from three blinded medical experts. Our pipeline achieved high accuracy, detecting liver cirrhosis with 100% sensitivity and 96% specificity. High sensitivities and specificities were also yielded for detecting ascites (95%, 95%), confusion (76%, 94%), abdominal pain (84%, 97%), and shortness of breath (87%, 97%) using the 70 billion parameter model, which outperformed smaller versions. Our study successfully demonstrates the capability of locally deployed LLMs to extract clinical information from free text with low hardware requirements.
Authors
- Daniel Truhn (ORCID: https://orcid.org/0000-0002-9605-0728)
- Jens Kleesiek (ORCID: https://orcid.org/0000-0001-8686-0682)
- Isabella C. Wiest (ORCID: https://orcid.org/0009-0004-4848-614X)
- Jiefu Zhu (ORCID: https://orcid.org/0009-0004-5765-5750)
- Marko van Treeck
- Dyke Ferber (ORCID: https://orcid.org/0009-0006-6195-9276)
- Zunamys I. Carrero (ORCID: https://orcid.org/0000-0001-8501-1566)
- Jakob Nikolas Kather (ORCID: https://orcid.org/0000-0002-3730-5348)
- Daniel Paech (ORCID: https://orcid.org/0000-0001-5755-6833)
- Radhika Juglan
- Matthias Ebert (ORCID: https://orcid.org/0000-0001-5713-5465)
- Sonja K. Meyer
Institutions
- European Molecular Biology Organization (DE)
- German Cancer Research Center (DE)
- TU Dortmund University (DE)
- Heidelberg University (DE)
- University Hospital Bonn (DE)
- University Hospital Heidelberg (DE)
- Institut für Medizinische Informatik, Biometrie und Epidemiologie (DE)
- National Center for Tumor Diseases (DE)
- Universitätsklinikum Aachen (DE)
- Universitätsklinikum Würzburg (DE)
- University Medical Centre Mannheim (DE)
- West German Heart and Vascular Center Essen (DE)
- University Hospital Carl Gustav Carus (DE)
Publication Details
- Journal
- npj Digital Medicine
- Published
- 2024-09-20
- DOI
- https://doi.org/10.1038/s41746-024-01233-2
- Citations
- 82
- Primary Topic
- Machine Learning in Healthcare
- Type
- article
- Field-Weighted Citation Impact
- 28.49
Funders
- National Institute for Health and Care Research
- Department of Health and Social Care
- European Commission
- Deutscher Akademischer Austauschdienst
- Bundesministerium für Bildung und Forschung
- Deutsche Krebshilfe
- Gemeinsame Bundesausschuss
- National Institutes of Health
- Horizon 2020 Framework Programme
- HORIZON EUROPE Framework Programme
- Leeds Biomedical Research Centre