Retrieval augmented generation for 10 large language models and its generalizability in assessing medical fitness
Large Language Models (LLMs) hold promise for medical applications but often lack domain-specific expertise. Retrieval Augmented Generation (RAG) enables customization by integrating specialized knowledge. This study assessed the accuracy, consistency, and safety of LLM-RAG models in determining surgical fitness and delivering preoperative instructions using 35 local and 23 international guidelines. Ten LLMs (e.g., GPT3.5, GPT4, GPT4o, Gemini, Llama2, and Llama3, Claude) were tested across 14 clinical scenarios. A total of 3234 responses were generated and compared to 448 human-generated answers. The GPT4 LLM-RAG model with international guidelines generated answers within 20 s and achieved the highest accuracy, which was significantly better than human-generated responses (96.4% vs. 86.6%, p = 0.016). Additionally, the model exhibited an absence of hallucinations and produced more consistent output than humans. This study underscores the potential of GPT-4-based LLM-RAG models to deliver highly accurate, efficient, and consistent preoperative assessments.
Authors
- Jasmine Chiat Ling Ong (ORCID: https://orcid.org/0000-0001-6916-5960)
- Hairil Rizal Abdullah (ORCID: https://orcid.org/0000-0003-1916-0832)
- Daniel Shu Wei Ting (ORCID: https://orcid.org/0000-0003-2264-7174)
- Chang‐Fu Kuo (ORCID: https://orcid.org/0000-0002-9770-5730)
- Shaochun Wu (ORCID: https://orcid.org/0000-0002-8035-1921)
- Vesela Kovacheva (ORCID: https://orcid.org/0000-0001-7367-3394)
- Joshua Yi Min Tung (ORCID: https://orcid.org/0000-0003-4991-9790)
- Kabilan Elangovan (ORCID: https://orcid.org/0000-0002-7711-7368)
- Nan Liu (ORCID: https://orcid.org/0000-0003-3610-4883)
- Liyuan Jin (ORCID: https://orcid.org/0000-0001-6738-1424)
- Alex Tiong Heng Sia (ORCID: https://orcid.org/0000-0003-3933-1299)
- Chai Rick Soh (ORCID: https://orcid.org/0000-0001-8756-7971)
- Yu He Ke
Institutions
- Brigham and Women's Hospital (US)
- Harvard University (US)
- National Sun Yat-sen University (TW)
- SingHealth (SG)
- Chang Gung University (TW)
- Singapore General Hospital (SG)
- Singapore National Eye Center (SG)
- Chang Gung Memorial Hospital (TW)
- Kaohsiung Chang Gung Memorial Hospital (TW)
- KK Women's and Children's Hospital (SG)
- Singapore Eye Research Institute (SG)
- Duke-NUS Medical School (SG)
Publication Details
- Journal
- npj Digital Medicine
- Published
- 2025-04-05
- DOI
- https://doi.org/10.1038/s41746-025-01519-z
- Citations
- 76
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 49.73
Funders
- Singapore General Hospital