Comparative evaluation and performance of large language models on expert level critical care questions: a benchmark study
LLMs exhibit exceptional accuracy and consistency, with four outperforming human physicians on a European-level practice exam. GPT-4o led in performance but raised concerns about energy consumption. Despite their potential in critical care, all models produced consistently incorrect answers, highlighting the need for more thorough and ongoing evaluations to guide responsible implementation in clinical settings.
Authors
- Davy van de Sande (ORCID: https://orcid.org/0000-0003-4484-0995)
- Sumesh Arora (ORCID: https://orcid.org/0000-0002-2065-3522)
- Michel E. van Genderen (ORCID: https://orcid.org/0000-0001-5668-3435)
- Marco Goeijenbier (ORCID: https://orcid.org/0000-0002-3165-8193)
- Jessica D. Workum (ORCID: https://orcid.org/0000-0002-4134-0242)
- Diederik A M P J Gommers (ORCID: https://orcid.org/0000-0001-6808-7702)
- Bas W. S. Volkers
Institutions
- Erasmus MC (NL)
- Spaarne Ziekenhuis (NL)
- Prince of Wales Hospital (AU)
- Elisabeth-TweeSteden Ziekenhuis (NL)
- Erasmus University Rotterdam (NL)
Publication Details
- Journal
- Critical Care
- Published
- 2025-02-10
- DOI
- https://doi.org/10.1186/s13054-025-05302-0
- Citations
- 22
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 14.40