Failure Modes of Prediction Models Built from Electronic Health Record Data: Methods and Preliminary Results of the First Pass of a Scoping Review

This report records the methods and preliminary results of the first pass of a scoping review of failure modes of prediction models built from electronic health record (EHR) data. A failure mode is a distinct way in which a failure can arise at any stage of a prediction project. A codebook of 80 anticipated failure modes in 14 families was built from three independent sources: a process map derived from a draft theory of how EHR data are created and reused, reporting standards and appraisal tools, and a random sample of the literature. PubMed was searched for records added on or before 23 September 2026, and 340 records were screened on title and abstract by one reviewer with advisory labels from a large language model. In all, 192 papers were provisionally included (86 central, 32 peripheral and 74 on mitigation). They mention 53 of the 80 anticipated failure modes, 47 of them in a central paper, and suggest seven provisional new modes. An earlier mapping of the failure modes onto the draft process map suggested that it could explain most of them at least in part and identified where it falls short. All results are preliminary and will be revised when the full texts are read in dedicated reviews of each family.

Authors

Publication Details

Journal
Open Science Framework
Published
2026-11-12
DOI
https://doi.org/10.17605/osf.io/3vr8j
Primary Topic
Machine Learning in Healthcare
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Failure Modes of Prediction Models Built from Electronic Health Record Data: Methods and Preliminary Results of the First Pass of a Scoping Review

Uday Pratap Singh
Open Science Framework
Machine Learning in Healthcare
article

Failure Modes of Prediction Models Built from Electronic Health Record Data: Methods and Preliminary Results of the First Pass of a Scoping Review

Uday Pratap Singh
article en

Abstract

This report records the methods and preliminary results of the first pass of a scoping review of failure modes of prediction models built from electronic health record (EHR) data. A failure mode is a distinct way in which a failure can arise at any stage of a prediction project. A codebook of 80 anticipated failure modes in 14 families was built from three independent sources: a process map derived from a draft theory of how EHR data are created and reused, reporting standards and appraisal tools, and a random sample of the literature. PubMed was searched for records added on or before 23 September 2026, and 340 records were screened on title and abstract by one reviewer with advisory labels from a large language model. In all, 192 papers were provisionally included (86 central, 32 peripheral and 74 on mitigation). They mention 53 of the 80 anticipated failure modes, 47 of them in a central paper, and suggest seven provisional new modes. An earlier mapping of the failure modes onto the draft process map suggested that it could explain most of them at least in part and identified where it falls short. All results are preliminary and will be revised when the full texts are read in dedicated reviews of each family.

Open Science Framework
Good health and well-being
Machine Learning in Healthcare
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon