Role of Surgeon-Model Interaction in the Real-World Performance of Surgical Risk Prediction Tools

BACKGROUND: Surgical risk models are typically evaluated independently of how surgeons interpret their predictions after deployment. We evaluated surgeon-model disagreement across distinct clinical environments and whether surgeon responses added prognostic information. STUDY DESIGN: In a multi-institutional study, surgeons reviewed 30-day complication risks generated by locally deployed NSQIP-SRC, POTTER, or MySurgeryRisk models and assessed risk relative to model estimates. We evaluated disagreement, observed outcomes, and model discrimination overall, among discordant assessments, and across model-specific risk quartiles. Logistic regression models incorporating model predictions and surgeon responses underwent 5-fold cross-validation, with discrimination compared using DeLong's test. RESULTS: Thirty-one surgeons evaluated 1,702 predictions across 2 academic medical centers. Surgeons disagreed with 53% of predictions, assessing risk as higher in 37% and lower in 16%. Lower-than-model assessments were followed by no complication in 84% of cases, whereas higher-than-model assessments were followed by a complication in 23%. Model AUCs were 0.66, 0.82, and 0.83 for NSQIP-SRC, POTTER, and MySurgeryRisk, respectively. Among discordant assessments, AUCs were 0.61, 0.79, and 0.83, respectively. Incorporating surgeon responses increased AUCs to 0.68, 0.83, and 0.86, respectively; none of these differences were statistically significant. Effects varied across risk quartiles. CONCLUSIONS: Surgeons frequently disagreed with deployed surgical risk models, but disagreement was not uniformly beneficial or consistently associated with poorer model discrimination. Evaluating how clinicians interpret and modify model predictions may be important for understanding real-world performance beyond conventional model metrics.

Authors

Institutions

Publication Details

Journal
Journal of the American College of Surgeons
Published
2026-11-18
DOI
https://doi.org/10.1097/xcs.0000000000002296
Primary Topic
Cardiac, Anesthesia and Surgical Outcomes
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Role of Surgeon-Model Interaction in the Real-World Performance of Surgical Risk Prediction Tools

Margaret T. Berrigan, Zeynep Bilgili, Anupamaa J. Seshadri, Azra Bihorac et al.
1 links
Journal of the American College of Surgeons
Cardiac, Anesthesia and Surgical Outcomes
article

Role of Surgeon-Model Interaction in the Real-World Performance of Surgical Risk Prediction Tools

Margaret T. Berrigan, Zeynep Bilgili, Anupamaa J. Seshadri, Azra Bihorac, Jayson S. Marwaha, Gabriel A. Brat, Charles Cook, Tyler John Loftus, Sofia Volpi, Stephen R. Odom, Alok Gupta, Charles S. Parsons, Daniel Bent
article en

Abstract

BACKGROUND: Surgical risk models are typically evaluated independently of how surgeons interpret their predictions after deployment. We evaluated surgeon-model disagreement across distinct clinical environments and whether surgeon responses added prognostic information. STUDY DESIGN: In a multi-institutional study, surgeons reviewed 30-day complication risks generated by locally deployed NSQIP-SRC, POTTER, or MySurgeryRisk models and assessed risk relative to model estimates. We evaluated disagreement, observed outcomes, and model discrimination overall, among discordant assessments, and across model-specific risk quartiles. Logistic regression models incorporating model predictions and surgeon responses underwent 5-fold cross-validation, with discrimination compared using DeLong's test. RESULTS: Thirty-one surgeons evaluated 1,702 predictions across 2 academic medical centers. Surgeons disagreed with 53% of predictions, assessing risk as higher in 37% and lower in 16%. Lower-than-model assessments were followed by no complication in 84% of cases, whereas higher-than-model assessments were followed by a complication in 23%. Model AUCs were 0.66, 0.82, and 0.83 for NSQIP-SRC, POTTER, and MySurgeryRisk, respectively. Among discordant assessments, AUCs were 0.61, 0.79, and 0.83, respectively. Incorporating surgeon responses increased AUCs to 0.68, 0.83, and 0.86, respectively; none of these differences were statistically significant. Effects varied across risk quartiles. CONCLUSIONS: Surgeons frequently disagreed with deployed surgical risk models, but disagreement was not uniformly beneficial or consistently associated with poorer model discrimination. Evaluating how clinicians interpret and modify model predictions may be important for understanding real-world performance beyond conventional model metrics.

Journal of the American College of Surgeons
Beth Israel Deaconess Medical Center (US), Harvard University (US), University of Michigan (US), University of Florida Health (US), University of Florida (US), Florida College (US)
Good health and well-being
Cardiac, Anesthesia and Surgical Outcomes
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon