Longitudinal Model Shifts of Machine Learning–Based Clinical Risk Prediction Models: Evaluation Study of Multiple Use Cases Across Different Hospitals

Clinical risk prediction models were affected by the dynamic and continuous evolution of clinical practices and workflows. The performance of the models evaluated in this study appeared stable when assessed using AUROCs, showing no significant variations over the years. Additional model shift investigations suggested that a calibration shift was present for certain use cases (delirium and sepsis). However, these changes did not have any impact on the clinical utility of the models based on DCA. Consequently, it is crucial to closely monitor data changes and detect possible model shifts, along with their potential influence on clinical decision-making.

Authors

Institutions

Publication Details

Journal
Journal of Medical Internet Research
Published
2024-12-13
DOI
https://doi.org/10.2196/51409
Citations
7
Primary Topic
Sepsis Diagnosis and Treatment
Type
article
Field-Weighted Citation Impact
2.71

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Longitudinal Model Shifts of Machine Learning–Based Clinical Risk Prediction Models: Evaluation Study of Multiple Use Cases Across Different Hospitals

Hong Sun, Kristof Depraetere, Fried-Michael Dahlweid, Jörg Stieg et al.
7 citations
Journal of Medical Internet Research
Sepsis Diagnosis and Treatment
2.71
article

Longitudinal Model Shifts of Machine Learning–Based Clinical Risk Prediction Models: Evaluation Study of Multiple Use Cases Across Different Hospitals

Hong Sun, Kristof Depraetere, Fried-Michael Dahlweid, Jörg Stieg, Janis Fliegenschmidt, Patricia Cabanillas Silva, Nikolai Hulde, Mohamed Rezk, Ralph Szymanowsky, Diana M. Roccaro-Waldmeyer, Laurent Meesseman, Vera von Dossow
article en
7 citations

Abstract

Clinical risk prediction models were affected by the dynamic and continuous evolution of clinical practices and workflows. The performance of the models evaluated in this study appeared stable when assessed using AUROCs, showing no significant variations over the years. Additional model shift investigations suggested that a calibration shift was present for certain use cases (delirium and sepsis). However, these changes did not have any impact on the clinical utility of the models based on DCA. Consequently, it is crucial to closely monitor data changes and detect possible model shifts, along with their potential influence on clinical decision-making.

Journal of Medical Internet ResearchVol. 26
Heart and Diabetes Center North Rhine-Westphalia (DE), Jiaxing University (CN), Ruhr University Bochum (DE)
European Commission, Ruhr-Universität Bochum
Good health and well-being
Openalex Percentile: Top 9%
Sepsis Diagnosis and Treatment
2.71
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.