Autonomous medical evaluation for guideline adherence of large language models
Autonomous Medical Evaluation for Guideline Adherence (AMEGA) is a comprehensive benchmark designed to evaluate large language models' adherence to medical guidelines across 20 diagnostic scenarios spanning 13 specialties. It includes an evaluation framework and methodology to assess models' capabilities in medical reasoning, differential diagnosis, treatment planning, and guideline adherence, using open-ended questions that mirror real-world clinical interactions. It includes 135 questions and 1337 weighted scoring elements designed to assess comprehensive medical knowledge. In tests of 17 LLMs, GPT-4 scored highest with 41.9/50, followed closely by Llama-3 70B and WizardLM-2-8x22B. For comparison, a recent medical graduate scored 25.8/50. The benchmark introduces novel content to avoid the issue of LLMs memorizing existing medical data. AMEGA's publicly available code supports further research in AI-assisted clinical decision-making, aiming to enhance patient care by aiding clinicians in diagnosis and treatment under time constraints.
Authors
- Lisa C. Adams (ORCID: https://orcid.org/0000-0001-5836-4542)
- Keno K. Bressem (ORCID: https://orcid.org/0000-0001-9249-8624)
- Daniel Truhn (ORCID: https://orcid.org/0000-0002-9605-0728)
- Robert Siepmann (ORCID: https://orcid.org/0000-0002-4608-5180)
- Felix Busch (ORCID: https://orcid.org/0000-0001-9770-8555)
- Marcus R. Makowski (ORCID: https://orcid.org/0000-0001-8778-647X)
- Nadine Bayerl (ORCID: https://orcid.org/0000-0002-8530-6783)
- Marc Huppertz (ORCID: https://orcid.org/0009-0004-2801-3679)
- Alexander Löser (ORCID: https://orcid.org/0000-0002-4440-3261)
- Philipp Prucker (ORCID: https://orcid.org/0000-0002-6968-2156)
- Conor Fallon
- Dennis Fast
Institutions
- Berliner Hochschule für Technik (DE)
- Friedrich-Alexander-Universität Erlangen-Nürnberg (DE)
- TUM Klinikum (DE)
- Universitätsklinikum Erlangen (DE)
- Universitätsklinikum Aachen (DE)
- Deutsches Herzzentrum München (DE)
- Technical University of Munich (DE)
Publication Details
- Journal
- npj Digital Medicine
- Published
- 2024-12-12
- DOI
- https://doi.org/10.1038/s41746-024-01356-6
- Citations
- 19
- Primary Topic
- Machine Learning in Healthcare
- Type
- article
- Field-Weighted Citation Impact
- 6.60
Funders
- Australian Government
- European Commission
- Deutsche Forschungsgemeinschaft
- Bundesministerium für Bildung und Forschung
- Technische Universität München