Comparison of machine learning methods and standard logistic regression to improve inpatient quality measurement in two clinical use cases

Background To evaluate whether machine learning (ML) methods (Elastic Net (EN), eXtreme Gradient Boosting (XGBoost), Feed Forward Neural Net (FNN)) can improve claims-based inpatient quality measurement by Logistic Regression. Methods This retrospective cohort study used German claims data from the years 2015-2021. The study population encompassed inpatient cases of acute myocardial infarction ( n = 165,130) and proximal humerus fracture ( n = 34,912), for which quality related outcomes were assessed. The performances of risk adjustment models based on machine learning methods (EN, XGBoost, FNN) were compared to stepwise backwards Logistic Regression by Receiver Operating Characteristics-Area under the Curve (ROC-AUC), Precision Recall-Area under the Curve (PR-AUC), Brier Score (BS). The institution-specific quality was measured by Standardised Mortality Ratios (SMR) which were used to visualise the impact of the tested methods on quality assessment. Results For most of the outcomes none or only marginal gains were found for the machine learning methods. Highest gain in model performance showed the FNN in comparison to Logistic Regression with a gain in ROC-AUC of 2.4%, in PR-AUC of 4.5%, and slightly in the BS with a loss of 0.007. The FNN was followed by XGBoost with a gain in ROC-AUC of 2.3%, anyhow this improvement was not reflected in a lower BS. Conclusions None of the machine learning methods tested is generally superior for creating quality indicators. Marginal gain in model performance should not be the main basis for choosing an adequate method; instead, interpretability should be emphasised, especially when dealing with new datasets with little knowledge of important risk factors.

Authors

Institutions

Publication Details

Journal
Research Methods in Medicine & Health Sciences
Published
2025-06-08
DOI
https://doi.org/10.1177/26320843251344587
Primary Topic
Advanced Statistical Process Monitoring
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Comparison of machine learning methods and standard logistic regression to improve inpatient quality measurement in two clinical use cases

Olaf Schoffer, Simone Wesselmann, Pompiliu Piso, Christian Günster et al.
Research Methods in Medicine & Health Sciences
Advanced Statistical Process Monitoring
article

Comparison of machine learning methods and standard logistic regression to improve inpatient quality measurement in two clinical use cases

Olaf Schoffer, Simone Wesselmann, Pompiliu Piso, Christian Günster, Tobias Helfen, Caroline Lang, Nils Sommer, Christoph Reißfelder, Christos Krogias, Thomas Datzmann, Ekkehard Schuler, Melissa Spoden, Christoph Gumbinger, Klaus‐Dieter Schaser, Elisa Henke, Patrik Dröge, Jochen Schmitt, Jessica Barlinn, Uwe Zeymer, Jörg Nowotny, J. Christoph Katthagen
article en

Abstract

Background To evaluate whether machine learning (ML) methods (Elastic Net (EN), eXtreme Gradient Boosting (XGBoost), Feed Forward Neural Net (FNN)) can improve claims-based inpatient quality measurement by Logistic Regression. Methods This retrospective cohort study used German claims data from the years 2015-2021. The study population encompassed inpatient cases of acute myocardial infarction ( n = 165,130) and proximal humerus fracture ( n = 34,912), for which quality related outcomes were assessed. The performances of risk adjustment models based on machine learning methods (EN, XGBoost, FNN) were compared to stepwise backwards Logistic Regression by Receiver Operating Characteristics-Area under the Curve (ROC-AUC), Precision Recall-Area under the Curve (PR-AUC), Brier Score (BS). The institution-specific quality was measured by Standardised Mortality Ratios (SMR) which were used to visualise the impact of the tested methods on quality assessment. Results For most of the outcomes none or only marginal gains were found for the machine learning methods. Highest gain in model performance showed the FNN in comparison to Logistic Regression with a gain in ROC-AUC of 2.4%, in PR-AUC of 4.5%, and slightly in the BS with a loss of 0.007. The FNN was followed by XGBoost with a gain in ROC-AUC of 2.3%, anyhow this improvement was not reflected in a lower BS. Conclusions None of the machine learning methods tested is generally superior for creating quality indicators. Marginal gain in model performance should not be the main basis for choosing an adequate method; instead, interpretability should be emphasised, especially when dealing with new datasets with little knowledge of important risk factors.

Research Methods in Medicine & Health SciencesVol. 7(2)
Heidelberg University (DE), University Hospital Bonn (DE), University Hospitals of the Ruhr-University of Bochum (DE), University Hospital Heidelberg (DE), LMU Klinikum (DE), University Hospital Münster (DE), Deutsche Gesellschaft für Gefäßchirurgie und Gefäßmedizin (DE), Deutsche Gesellschaft für Hämatologie und Medizinische Onkologie (DE), Cook Research (United States) (US), Krankenhaus Barmherzige Brüder (DE), Klinikum Ludwigshafen (DE), Helios Kliniken (DE), University Medical Centre Mannheim (DE), University Hospital Carl Gustav Carus (DE), Technische Universität Dresden (DE), Ludwig-Maximilians-Universität München (DE)
Openalex Percentile: Top 87%
Advanced Statistical Process Monitoring
DOISourceOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.