"No Errors Found'': Review Verdicts Shift the Threshold at Which Language Models Flag Clinical Errors

Full code repository for the No Errors Found project. Layout Path Contents Top level run_experiment.py Pipeline entry point (steps below) analysis_v3.py Cross-model analysis entry point dump_generation_config.py Effective decoding settings and software versions quick_check.py Per-header summary of one model's raw answers config/ (every setting of the study) paths.py Data and results folders steps.py Default steps datasets.py MEDEC-MS and MedErrBench-EN files, 739-text sample size conditions.py Free-text header conditions and wordings structured.py FHIR, flat JSON and HL7 v2 status blocks factorial.py Verdict x reviewer conditions prompts.py System prompt, user template and prompt variants studies.py Study components: conditions x variants x texts x models models.py Model registry, decoding and serving settings inference.py GPU, server, pre-check and request settings evaluation.py Bootstrap, non-responsive rule and report settings noerrorsfound/pipeline/ common.py Output paths, logging, command-line parsing selection.py Model and study selection, GPU assignment dataset.py Benchmark loading, sentence parsing, case linking prompts.py Prompt rendering, content-hashed prompt registry (build step) client.py OpenAI-compatible requests, token log-probabilities server.py vLLM server start, health checks, shutdown hardware.py GPU memory, GPU processes, disk checks precheck.py Pre-run check of every model (readable answers, truncation, speed) inference.py Resumable inference, parallel GPU workers (infer step) parsing.py JSON answer parsing, repetition-loop detection metrics.py Sensitivity, specificity, d', c, bootstrap deltas, McNemar, Holm correction.py Secondary correction-quality scoring predictions.py Predictions, fingerprints, GEE summary, paired subsets evaluate.py Per-model effects and tests (evaluate step) diagnose.py Answer-format diagnostics (diagnose step) meta.py Random-effects pooling and meta-regression results.py Loader for all per-model results tables.py Report tables style.py Figure style figures.py Report figures report.py Report, index and run manifest (report step) steps.py Step dispatcher noerrorsfound/analysis/ common.py Loading, model metadata, non-responsive rule effects.py Paired bootstrap effects parse_sensitivity.py Unreadable-answer scoring sensitivity discrimination.py AUROC of P(error_flag = 1) localization.py Sentence localization and matched-hit analysis zroc.py zROC slopes noise_floor.py Rerun flip rates and excess flips metaregression.py REML, HKSJ pooling, meta-regression ablation.py 739-text comparisons across all models structured.py Structured-metadata effects variants.py Prompt-variant effects factorial.py Verdict x reviewer effects compile.py Model table, groups, pooled and primary tests robustness.py Error-type restriction, reasoning length, model pairs glmm.py Pooled case-clustered logistic model export.py Slim prediction files and summary main.py Analysis driver Other noerrorsfound/environment.py Generation-config and environment dump chat_templates/ Llama 3 chat template for OpenBioLLM-8B Data Clone the two benchmarks into data/: git clone https://github.com/abachaa/MEDEC data/MEDEC git clone https://github.com/congboma/MedErrBench data/MedErrBench Environment Linux with NVIDIA GPUs (two 48 GB GPUs for the 27B and 32B models). pip install -r requirements.txt Contact Nahid Hossain, MSc ([email protected]) Md Faisal Kabir, PhD ([email protected]) School of Science, Engineering and Technology The Pennsylvania State University, Harrisburg, PA, USA License The source code in this repository is released under the MIT License. Result tables and model predictions are released under CC BY 4.0. The MEDEC and MedErrBench datasets are not included in this repository and remain subject to their original licenses and terms of use. Disclaimer This project is intended for research purposes only and is not a medical device. It should not be used for clinical decision-making.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-13
DOI
https://doi.org/10.5281/zenodo.24183239
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

"No Errors Found'': Review Verdicts Shift the Threshold at Which Language Models Flag Clinical Errors

Md Faisal Kabir, Nahid Hossain
4 links
Zenodo (CERN European Organization for Nuclear Research)
Artificial Intelligence in Healthcare and Education
article

"No Errors Found'': Review Verdicts Shift the Threshold at Which Language Models Flag Clinical Errors

Md Faisal Kabir, Nahid Hossain
article en

Abstract

Full code repository for the No Errors Found project. Layout Path Contents Top level run_experiment.py Pipeline entry point (steps below) analysis_v3.py Cross-model analysis entry point dump_generation_config.py Effective decoding settings and software versions quick_check.py Per-header summary of one model's raw answers config/ (every setting of the study) paths.py Data and results folders steps.py Default steps datasets.py MEDEC-MS and MedErrBench-EN files, 739-text sample size conditions.py Free-text header conditions and wordings structured.py FHIR, flat JSON and HL7 v2 status blocks factorial.py Verdict x reviewer conditions prompts.py System prompt, user template and prompt variants studies.py Study components: conditions x variants x texts x models models.py Model registry, decoding and serving settings inference.py GPU, server, pre-check and request settings evaluation.py Bootstrap, non-responsive rule and report settings noerrorsfound/pipeline/ common.py Output paths, logging, command-line parsing selection.py Model and study selection, GPU assignment dataset.py Benchmark loading, sentence parsing, case linking prompts.py Prompt rendering, content-hashed prompt registry (build step) client.py OpenAI-compatible requests, token log-probabilities server.py vLLM server start, health checks, shutdown hardware.py GPU memory, GPU processes, disk checks precheck.py Pre-run check of every model (readable answers, truncation, speed) inference.py Resumable inference, parallel GPU workers (infer step) parsing.py JSON answer parsing, repetition-loop detection metrics.py Sensitivity, specificity, d', c, bootstrap deltas, McNemar, Holm correction.py Secondary correction-quality scoring predictions.py Predictions, fingerprints, GEE summary, paired subsets evaluate.py Per-model effects and tests (evaluate step) diagnose.py Answer-format diagnostics (diagnose step) meta.py Random-effects pooling and meta-regression results.py Loader for all per-model results tables.py Report tables style.py Figure style figures.py Report figures report.py Report, index and run manifest (report step) steps.py Step dispatcher noerrorsfound/analysis/ common.py Loading, model metadata, non-responsive rule effects.py Paired bootstrap effects parse_sensitivity.py Unreadable-answer scoring sensitivity discrimination.py AUROC of P(error_flag = 1) localization.py Sentence localization and matched-hit analysis zroc.py zROC slopes noise_floor.py Rerun flip rates and excess flips metaregression.py REML, HKSJ pooling, meta-regression ablation.py 739-text comparisons across all models structured.py Structured-metadata effects variants.py Prompt-variant effects factorial.py Verdict x reviewer effects compile.py Model table, groups, pooled and primary tests robustness.py Error-type restriction, reasoning length, model pairs glmm.py Pooled case-clustered logistic model export.py Slim prediction files and summary main.py Analysis driver Other noerrorsfound/environment.py Generation-config and environment dump chat_templates/ Llama 3 chat template for OpenBioLLM-8B Data Clone the two benchmarks into data/: git clone https://github.com/abachaa/MEDEC data/MEDEC git clone https://github.com/congboma/MedErrBench data/MedErrBench Environment Linux with NVIDIA GPUs (two 48 GB GPUs for the 27B and 32B models). pip install -r requirements.txt Contact Nahid Hossain, MSc ([email protected]) Md Faisal Kabir, PhD ([email protected]) School of Science, Engineering and Technology The Pennsylvania State University, Harrisburg, PA, USA License The source code in this repository is released under the MIT License. Result tables and model predictions are released under CC BY 4.0. The MEDEC and MedErrBench datasets are not included in this repository and remain subject to their original licenses and terms of use. Disclaimer This project is intended for research purposes only and is not a medical device. It should not be used for clinical decision-making.

Zenodo (CERN European Organization for Nuclear Research)
Pennsylvania State University (US)
Good health and well-being
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon