Role of Surgeon-Model Interaction in the Real-World Performance of Surgical Risk Prediction Tools
BACKGROUND: Surgical risk models are typically evaluated independently of how surgeons interpret their predictions after deployment. We evaluated surgeon-model disagreement across distinct clinical environments and whether surgeon responses added prognostic information. STUDY DESIGN: In a multi-institutional study, surgeons reviewed 30-day complication risks generated by locally deployed NSQIP-SRC, POTTER, or MySurgeryRisk models and assessed risk relative to model estimates. We evaluated disagreement, observed outcomes, and model discrimination overall, among discordant assessments, and across model-specific risk quartiles. Logistic regression models incorporating model predictions and surgeon responses underwent 5-fold cross-validation, with discrimination compared using DeLong's test. RESULTS: Thirty-one surgeons evaluated 1,702 predictions across 2 academic medical centers. Surgeons disagreed with 53% of predictions, assessing risk as higher in 37% and lower in 16%. Lower-than-model assessments were followed by no complication in 84% of cases, whereas higher-than-model assessments were followed by a complication in 23%. Model AUCs were 0.66, 0.82, and 0.83 for NSQIP-SRC, POTTER, and MySurgeryRisk, respectively. Among discordant assessments, AUCs were 0.61, 0.79, and 0.83, respectively. Incorporating surgeon responses increased AUCs to 0.68, 0.83, and 0.86, respectively; none of these differences were statistically significant. Effects varied across risk quartiles. CONCLUSIONS: Surgeons frequently disagreed with deployed surgical risk models, but disagreement was not uniformly beneficial or consistently associated with poorer model discrimination. Evaluating how clinicians interpret and modify model predictions may be important for understanding real-world performance beyond conventional model metrics.
Authors
- Margaret T. Berrigan (ORCID: https://orcid.org/0009-0009-4538-3459)
- Zeynep Bilgili
- Anupamaa J. Seshadri (ORCID: https://orcid.org/0000-0001-8432-7518)
- Azra Bihorac (ORCID: https://orcid.org/0000-0002-5745-2863)
- Jayson S. Marwaha (ORCID: https://orcid.org/0000-0002-3833-7448)
- Gabriel A. Brat (ORCID: https://orcid.org/0000-0003-3928-5931)
- Charles Cook
- Tyler John Loftus (ORCID: https://orcid.org/0000-0001-5354-443X)
- Sofia Volpi
- Stephen R. Odom
- Alok Gupta
- Charles S. Parsons
- Daniel Bent
Institutions
- Beth Israel Deaconess Medical Center (US)
- Harvard University (US)
- University of Michigan (US)
- University of Florida Health (US)
- University of Florida (US)
- Florida College (US)
Publication Details
- Journal
- Journal of the American College of Surgeons
- Published
- 2026-11-18
- DOI
- https://doi.org/10.1097/xcs.0000000000002296
- Primary Topic
- Cardiac, Anesthesia and Surgical Outcomes
- Type
- article
- Field-Weighted Citation Impact
- 0.00