Evaluating accuracy and reproducibility of large language model performance on critical care assessments in pharmacy education
ChatGPT-4 was the most accurate LLM on critical care pharmacy questions and few-shot CoT improved accuracy the most. Average student accuracy was similar to LLMs overall, and higher on knowledge application questions. These findings support the need for future assessment of customized training for the type of output needed. Reliance on LLMs is only supported with recall-based questions.
Authors
- Susan Smith (ORCID: https://orcid.org/0000-0002-5171-8405)
- Brian Murray (ORCID: https://orcid.org/0000-0001-6660-9749)
- Andrea Sikora (ORCID: https://orcid.org/0000-0003-2020-0571)
- William Anthony Hawkins (ORCID: https://orcid.org/0000-0002-4083-7919)
- Mengxuan Hu (ORCID: https://orcid.org/0000-0002-8822-2884)
- Amoreena Most (ORCID: https://orcid.org/0000-0002-1958-5374)
- Sheng Li (ORCID: https://orcid.org/0000-0003-1205-8632)
- Huibo Yang
Institutions
- Georgia College & State University (US)
- University of Georgia (US)
- Augusta University (US)
- University of Virginia (US)
- University of Montana (US)
Publication Details
- Journal
- Frontiers in Artificial Intelligence
- Published
- 2025-01-09
- DOI
- https://doi.org/10.3389/frai.2024.1514896
- Citations
- 13
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 8.51