Benchmarking proprietary and open-source language and vision-language models for gastroenterology clinical reasoning

This study evaluated the effectiveness of large language models (LLMs) and vision-language models (VLMs) in gastroenterology. We used board-style multiple-choice questions to assess the performance of both proprietary and open-source LLMs and VLMs-including GPT, Claude, Gemini, Mistral, Llama, Mixtral, Phi, and Qwen, across different interfaces, computing environments, and levels of compression (quantization). Among the proprietary models, o1-preview (82.0%) and Claude3.5-Sonnet (74.0%) had the highest accuracy, outperforming the top open-source models: Llama3.3-70b (65.7%) and Qwen-2.5-72b (61.0%). Among the small quantized open-source models, the 8-bit Llama 3.2-11b (51.7%) and 6-bit Phi3-14b (48.7%) performed the best, with scores comparable to their full-precision counterparts. Notably, VLM accuracy on image-containing questions improved (~10%) when given human-generated captions, remained unchanged with original images, and declined with LLM-generated captions. Further research is warranted to evaluate model capabilities in real-world clinical decision-making scenarios.

Authors

Institutions

Publication Details

Journal
npj Digital Medicine
Published
2025-11-27
DOI
https://doi.org/10.1038/s41746-025-02174-0
Citations
1
Primary Topic
Multimodal Machine Learning Applications
Type
article
Field-Weighted Citation Impact
1.28

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Benchmarking proprietary and open-source language and vision-language models for gastroenterology clinical reasoning

Aasma Shaukat, Shuhaib Ali, Juan Echavarria, Sara Rafiee et al.
1 citations
npj Digital Medicine
Multimodal Machine Learning Applications
1.28
article

Benchmarking proprietary and open-source language and vision-language models for gastroenterology clinical reasoning

Aasma Shaukat, Shuhaib Ali, Juan Echavarria, Sara Rafiee, Omer Shahab, Bara El Kurdi, Girish N. Nadkarni, Jamil S. Samaan, Farah Ladak, Jamie O. Yang, Thomas Savage, Samuel Margolis, Seyed Amir Ahmad Safavi‐Naini, Nicholas P. Tatonetti, Ali Soroush, Zahra Shahhoseini, Reem Al Shabeeb, Sumbal Babar
article en
1 citations

Abstract

This study evaluated the effectiveness of large language models (LLMs) and vision-language models (VLMs) in gastroenterology. We used board-style multiple-choice questions to assess the performance of both proprietary and open-source LLMs and VLMs-including GPT, Claude, Gemini, Mistral, Llama, Mixtral, Phi, and Qwen, across different interfaces, computing environments, and levels of compression (quantization). Among the proprietary models, o1-preview (82.0%) and Claude3.5-Sonnet (74.0%) had the highest accuracy, outperforming the top open-source models: Llama3.3-70b (65.7%) and Qwen-2.5-72b (61.0%). Among the small quantized open-source models, the 8-bit Llama 3.2-11b (51.7%) and 6-bit Phi3-14b (48.7%) performed the best, with scores comparable to their full-precision counterparts. Notably, VLM accuracy on image-containing questions improved (~10%) when given human-generated captions, remained unchanged with original images, and declined with LLM-generated captions. Further research is warranted to evaluate model capabilities in real-world clinical decision-making scenarios.

npj Digital MedicineVol. 8(1)
Cedars-Sinai Medical Center (US), University of California, Los Angeles (US), The University of Texas at San Antonio Health Science Center (US), Inova Fairfax Hospital (US), University of California System (US), Child Health and Development Institute (US), Virginia Hospital Center (US), Texas Center for Infectious Disease (US), New York University (US), Shahid Beheshti University of Medical Sciences (IR), Columbia University (US), Stanford University (US), Icahn School of Medicine at Mount Sinai (US)
Amgen, American College of Gastroenterology, American Gastroenterological Association
Openalex Percentile: Top 14%
Multimodal Machine Learning Applications
1.28
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.