Explainable machine learning for leakage-aware risk stratification of electrocardiograph verification records

Metrological verification of electrocardiographs (ECGs) performance and safety generates structured technical and measurement data that may support the identification of devices requiring additional expert review. Machine-learning models may assist verification workflows by devices at elevated risk of failure. This study investigates the predictive value and limitations of routinely collected metrological verification data for identifying electrocardiographs at increased risk of verification failure. A retrospective analysis of 3,956 electrocardiograph verification records, including 3,870 passing and 86 failing records, was conducted. Technical and measurement variables available before the final verification decision were selected, while variables associated with potential data leakage were excluded. Three machine-learning algorithms, LightGBM, Balanced Random Forest, and SMOTE Random Forest, were compared using repeated stratified five-fold cross-validation, with PR-AUC as the primary model selection criterion. The selected model was further evaluated using out-of-fold predictions, bootstrap confidence intervals, calibration analysis, feature importance, and group-based validation to assess performance on previously unseen manufacturers and device models. Among the evaluated algorithms, LightGBM achieved the highest precision-recall performance, demonstrating substantial improvement over the baseline failure prevalence and revealing meaningful predictive information in routinely collected metrological data. Explainability analyses identified measurement-related characteristics and device-specific information as important predictors, while ablation analysis demonstrated that predictive discrimination was retained even after removing device identity. The study further demonstrated the value of combining leakage-controlled modelling, explainability, and robustness assessment to critically evaluate machine-learning performance in real-world medical device verification. Nevertheless, limited precision, poor probability calibration, and reduced performance on unseen manufacturers and device models highlighted important limitations, while decision curve analysis provided insufficient evidence of consistent operational benefit. This study demonstrates the potential of routinely collected metrological verification data to support AI-driven risk stratification, contributing to the broader transition toward data-driven medical device management advocated by the World Health Organization. While the findings establish a basis for further development of intelligent quality assurance systems, limitations in precision, calibration, and generalizability highlight the need for independent validation and demonstrated operational benefit before implementation in routine verification practice.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-19
DOI
https://doi.org/10.21203/rs.3.rs-10844417/v1
Primary Topic
Quality and Safety in Healthcare
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Explainable machine learning for leakage-aware risk stratification of electrocardiograph verification records

Ermin Begović, Berina Hasanefendić, Neslanović Adin, Almir Badnjević et al.
Research Square
Quality and Safety in Healthcare
preprint

Explainable machine learning for leakage-aware risk stratification of electrocardiograph verification records

Ermin Begović, Berina Hasanefendić, Neslanović Adin, Almir Badnjević, Irhad Neslanović
preprint en

Abstract

Metrological verification of electrocardiographs (ECGs) performance and safety generates structured technical and measurement data that may support the identification of devices requiring additional expert review. Machine-learning models may assist verification workflows by devices at elevated risk of failure. This study investigates the predictive value and limitations of routinely collected metrological verification data for identifying electrocardiographs at increased risk of verification failure. A retrospective analysis of 3,956 electrocardiograph verification records, including 3,870 passing and 86 failing records, was conducted. Technical and measurement variables available before the final verification decision were selected, while variables associated with potential data leakage were excluded. Three machine-learning algorithms, LightGBM, Balanced Random Forest, and SMOTE Random Forest, were compared using repeated stratified five-fold cross-validation, with PR-AUC as the primary model selection criterion. The selected model was further evaluated using out-of-fold predictions, bootstrap confidence intervals, calibration analysis, feature importance, and group-based validation to assess performance on previously unseen manufacturers and device models. Among the evaluated algorithms, LightGBM achieved the highest precision-recall performance, demonstrating substantial improvement over the baseline failure prevalence and revealing meaningful predictive information in routinely collected metrological data. Explainability analyses identified measurement-related characteristics and device-specific information as important predictors, while ablation analysis demonstrated that predictive discrimination was retained even after removing device identity. The study further demonstrated the value of combining leakage-controlled modelling, explainability, and robustness assessment to critically evaluate machine-learning performance in real-world medical device verification. Nevertheless, limited precision, poor probability calibration, and reduced performance on unseen manufacturers and device models highlighted important limitations, while decision curve analysis provided insufficient evidence of consistent operational benefit. This study demonstrates the potential of routinely collected metrological verification data to support AI-driven risk stratification, contributing to the broader transition toward data-driven medical device management advocated by the World Health Organization. While the findings establish a basis for further development of intelligent quality assurance systems, limitations in precision, calibration, and generalizability highlight the need for independent validation and demonstrated operational benefit before implementation in routine verification practice.

Research Square
University of Zenica (BA), Clinical Center University of Sarajevo (BA)
Quality and Safety in Healthcare
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon