Evaluating Locally Run Large Language Models (Gemma 2, Mistral Nemo, and Llama 3) for Outpatient Otorhinolaryngology Care: Retrospective Study

Although locally run LLM models still underperform compared with their web-based counterparts, they achieved respectable results on outpatient treatment in this study. Nevertheless, the retrospective and single-center nature of the study, along with the clinicians' documentation style, may have introduced bias in favor of human recommendations. In the future, locally run LLMs will help address data protection concerns; however, further refinement and prospective validation are still needed to meet strict medical device requirements. As locally run LLMs continue to evolve, they are likely to become comparably powerful to web-based LLMs and become established as useful tools to support doctors in clinical practice.

Authors

Institutions

Publication Details

Journal
JMIR Formative Research
Published
2025-11-25
DOI
https://doi.org/10.2196/76896
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Evaluating Locally Run Large Language Models (Gemma 2, Mistral Nemo, and Llama 3) for Outpatient Otorhinolaryngology Care: Retrospective Study

Katharina Bahr‐Hamm, Jonas Eckrich, Tilman Huppertz, Christoph Raphael Buhr et al.
JMIR Formative Research
Artificial Intelligence in Healthcare and Education
article

Evaluating Locally Run Large Language Models (Gemma 2, Mistral Nemo, and Llama 3) for Outpatient Otorhinolaryngology Care: Retrospective Study

Katharina Bahr‐Hamm, Jonas Eckrich, Tilman Huppertz, Christoph Raphael Buhr, Christopher Seifen, Sebastian Kühn, Johannes Pordzik, Andrew Blaikie, Harry A. Smith, Tom Kelsey, Christoph Matthias
article en

Abstract

Although locally run LLM models still underperform compared with their web-based counterparts, they achieved respectable results on outpatient treatment in this study. Nevertheless, the retrospective and single-center nature of the study, along with the clinicians' documentation style, may have introduced bias in favor of human recommendations. In the future, locally run LLMs will help address data protection concerns; however, further refinement and prospective validation are still needed to meet strict medical device requirements. As locally run LLMs continue to evolve, they are likely to become comparably powerful to web-based LLMs and become established as useful tools to support doctors in clinical practice.

JMIR Formative ResearchVol. 9
Philipps University of Marburg (DE), University of St Andrews (GB), Johannes Gutenberg University Mainz (DE), Universitätsklinikum Gießen und Marburg (DE), University Medical Center of the Johannes Gutenberg University Mainz (DE)
Openalex Percentile: Top 70%
Artificial Intelligence in Healthcare and Education
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.