Cross-institutional calibration and representation stability of language- model risk estimates in digestive surgery: a retrospective two-database study

Abstract Background Large language models can report numerical surgical risks, but discrimination, probability accuracy and stability under equivalent input representations may differ. We evaluated these properties across two institutions using locally deployed language models and common clinical predictors. Methods This retrospective study included 1,909 digestive resection operations in VitalDB and 1,791 scheduled-resection candidates in MOVER. The outcome was in-hospital death or discharge more than 14 days after anesthesia end. Qwen3-14B and Gemma3-12B, both quantized to Q4_K_M, generated probabilities from ten fields at anesthesia end. Logistic regression and gradient boosting used the same clinical fields. Source evaluation used patient-grouped folds; external evaluation transferred models and source-only logistic calibration maps without fitting to MOVER outcomes. Paired tests compared identical-text replay, reversed field order and equivalent hemoglobin/albumin unit conversion. Uncertainty was estimated with 1,000 patient-cluster bootstrap samples conditional on fitted models. Results The outcome occurred in 220/1,909 VitalDB operations (11.5%) and 140/1,791 MOVER candidates (7.8%). External areas under the receiver operating characteristic curve were 0.788 for logistic regression, 0.782 for gradient boosting, 0.656 for Qwen and 0.722 for Gemma. Raw Brier scores were 0.0760, 0.0723, 0.0699 and 0.0689, respectively. Source calibration increased external Brier scores by 0.01065 for Qwen (95% interval 0.00791–0.01320) and 0.00200 for Gemma (0.00015–0.00377). Among 429 MOVER operations with a convertible laboratory value, unit conversion changed raw risk by an average absolute 0.93 percentage points for Qwen (0.75–1.14) and 6.49 for Gemma (5.77–7.20). The corresponding changes after fixed source calibration were 1.20 and 2.66 percentage points. Some representation changes improved outcome error, whereas others worsened it. Conclusions In these two quantized models, lower aggregate probability error coexisted with weaker discrimination, unsuccessful calibration transfer and representation-dependent risks. External outcome evaluation and paired representation tests provide complementary evidence when assessing numerical language-model risk estimates.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-16
DOI
https://doi.org/10.21203/rs.3.rs-11262838/v1
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Cross-institutional calibration and representation stability of language- model risk estimates in digestive surgery: a retrospective two-database study

Zhi Qiaoming, 邵栋豪, Xiao Shi, ye han et al.
2 links
Research Square
Artificial Intelligence in Healthcare and Education
preprint

Cross-institutional calibration and representation stability of language- model risk estimates in digestive surgery: a retrospective two-database study

Zhi Qiaoming, 邵栋豪, Xiao Shi, ye han, shao dong hao
preprint en

Abstract

Abstract Background Large language models can report numerical surgical risks, but discrimination, probability accuracy and stability under equivalent input representations may differ. We evaluated these properties across two institutions using locally deployed language models and common clinical predictors. Methods This retrospective study included 1,909 digestive resection operations in VitalDB and 1,791 scheduled-resection candidates in MOVER. The outcome was in-hospital death or discharge more than 14 days after anesthesia end. Qwen3-14B and Gemma3-12B, both quantized to Q4_K_M, generated probabilities from ten fields at anesthesia end. Logistic regression and gradient boosting used the same clinical fields. Source evaluation used patient-grouped folds; external evaluation transferred models and source-only logistic calibration maps without fitting to MOVER outcomes. Paired tests compared identical-text replay, reversed field order and equivalent hemoglobin/albumin unit conversion. Uncertainty was estimated with 1,000 patient-cluster bootstrap samples conditional on fitted models. Results The outcome occurred in 220/1,909 VitalDB operations (11.5%) and 140/1,791 MOVER candidates (7.8%). External areas under the receiver operating characteristic curve were 0.788 for logistic regression, 0.782 for gradient boosting, 0.656 for Qwen and 0.722 for Gemma. Raw Brier scores were 0.0760, 0.0723, 0.0699 and 0.0689, respectively. Source calibration increased external Brier scores by 0.01065 for Qwen (95% interval 0.00791–0.01320) and 0.00200 for Gemma (0.00015–0.00377). Among 429 MOVER operations with a convertible laboratory value, unit conversion changed raw risk by an average absolute 0.93 percentage points for Qwen (0.75–1.14) and 6.49 for Gemma (5.77–7.20). The corresponding changes after fixed source calibration were 1.20 and 2.66 percentage points. Some representation changes improved outcome error, whereas others worsened it. Conclusions In these two quantized models, lower aggregate probability error coexisted with weaker discrimination, unsuccessful calibration transfer and representation-dependent risks. External outcome evaluation and paired representation tests provide complementary evidence when assessing numerical language-model risk estimates.

Research Square
First Affiliated Hospital of Soochow University (CN)
Good health and well-being
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon