BenchmarkHub: an open-source platform for reproducible evaluation of large language models in healthcare

Background: Large language models (LLMs) are solely judged by their scores on biomedical benchmarks, yet these scores are rarely reproducible since the prompts, inference settings and model builds are seldom reported. We developed BenchmarkHub, an open-source platform for auditable evaluation of language and vision-language models for biomedical research. Methods: BenchmarkHub is a Python web application (Django 5.2, SQLite) released under the MIT licence and available at https://github.com/juanjuanignacio/benchmarkhub. The application can be accessed through a browser interface, a set of command-line tools and a REST API, so interactive exploration and scripted pipelines produce identical, comparable records. The platform samples questions with fixed seeds, runs models directly or as tool-using agents in a code sandbox, grades answers with deterministic rules and stores every prompt, reply and tool call. The interface provides a leaderboard to rank performance of the models, allowing model comparison without any preexisting programming knowledge. Findings: We used BenchmarkHub to evaluate 12 open-weight models in five experiments on 15 biomedical benchmarks (11 text, two multimodal and two computational biology) and two general benchmarks. The pooled accuracy ranged from 46.1% (95% CI 43.0-49.3) to 80.9% (78.3-83.3). Re-grading the responses with simple answer-extraction rules lowered a model's accuracy by up to 33.1 percentage points, leading to a reorder of the leaderboard. Published scores were found for 23 out of the 143 benchmark/model combinations tested. The accuracy measured by BenchmarkHub was lower than in the published results in 19 out of the 23 available comparisons. The effect of four prompting protocols (1: default prompt; 2: asking the model to answer the letter only; 3: chain-of-thought prompting; 4: three-shot prompting) changed MedQA accuracy by up to 11.3 points but was not significant after multiple-testing correction with 150 questions. We also tested the impact of ablation on benchmarking using the MMMU benchmark dataset. Removing images lowered the accuracy by 18.0 and 19.3 points respectively for the two strongest models. Finally, the accuracy of agentic benchmarks using a sandboxed interpreter increased on computational biology tasks by 36-63 points. Interpretation: Benchmarking is essential to assess the impact and validity of LLMs for medical applications. BenchmarkHub allows a fair and rigorous comparison of models across established datasets, including the possibility to upload their own datasets. The suite includes visual tools to ease model comparison and ranking without any programming knowledge. Self-hosted infrastructure that records every choice and parameter makes benchmark claims checkable without data leaving the institution.

Authors

Institutions

Publication Details

Journal
medRxiv
Published
2026-11-13
DOI
https://doi.org/10.64898/2026.11.11.26367866
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
preprint

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

BenchmarkHub: an open-source platform for reproducible evaluation of large language models in healthcare

Daniel Jiménez‐Carretero, Fátima Sánchez‐Cabo, J I Alvarez-Arenas
1 links
medRxiv
Artificial Intelligence in Healthcare and Education
preprint

BenchmarkHub: an open-source platform for reproducible evaluation of large language models in healthcare

Daniel Jiménez‐Carretero, Fátima Sánchez‐Cabo, J I Alvarez-Arenas
preprint en

Abstract

Background: Large language models (LLMs) are solely judged by their scores on biomedical benchmarks, yet these scores are rarely reproducible since the prompts, inference settings and model builds are seldom reported. We developed BenchmarkHub, an open-source platform for auditable evaluation of language and vision-language models for biomedical research. Methods: BenchmarkHub is a Python web application (Django 5.2, SQLite) released under the MIT licence and available at https://github.com/juanjuanignacio/benchmarkhub. The application can be accessed through a browser interface, a set of command-line tools and a REST API, so interactive exploration and scripted pipelines produce identical, comparable records. The platform samples questions with fixed seeds, runs models directly or as tool-using agents in a code sandbox, grades answers with deterministic rules and stores every prompt, reply and tool call. The interface provides a leaderboard to rank performance of the models, allowing model comparison without any preexisting programming knowledge. Findings: We used BenchmarkHub to evaluate 12 open-weight models in five experiments on 15 biomedical benchmarks (11 text, two multimodal and two computational biology) and two general benchmarks. The pooled accuracy ranged from 46.1% (95% CI 43.0-49.3) to 80.9% (78.3-83.3). Re-grading the responses with simple answer-extraction rules lowered a model's accuracy by up to 33.1 percentage points, leading to a reorder of the leaderboard. Published scores were found for 23 out of the 143 benchmark/model combinations tested. The accuracy measured by BenchmarkHub was lower than in the published results in 19 out of the 23 available comparisons. The effect of four prompting protocols (1: default prompt; 2: asking the model to answer the letter only; 3: chain-of-thought prompting; 4: three-shot prompting) changed MedQA accuracy by up to 11.3 points but was not significant after multiple-testing correction with 150 questions. We also tested the impact of ablation on benchmarking using the MMMU benchmark dataset. Removing images lowered the accuracy by 18.0 and 19.3 points respectively for the two strongest models. Finally, the accuracy of agentic benchmarks using a sandboxed interpreter increased on computational biology tasks by 36-63 points. Interpretation: Benchmarking is essential to assess the impact and validity of LLMs for medical applications. BenchmarkHub allows a fair and rigorous comparison of models across established datasets, including the possibility to upload their own datasets. The suite includes visual tools to ease model comparison and ranking without any programming knowledge. Self-hosted infrastructure that records every choice and parameter makes benchmark claims checkable without data leaving the institution.

medRxiv
Spanish National Centre for Cardiovascular Research (ES)
Ministerio de Ciencia, Innovación y Universidades, European Commission, Centro Nacional de Investigaciones Cardiovasculares, Instituto de Salud Carlos III, Agencia Estatal de Investigación, NextGenerationEU, European Health and Digital Executive Agency
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon