Machine learning and logistic regression for ascertainable three-year recurrence status after soft tissue sarcoma surgery

BACKGROUNDPostoperative local-recurrence estimates could inform surveillance discussions after soft tissue sarcoma surgery, but machine-learning performance can be overstated when model selection is not separated from evaluation or when simpler regression models are not compared. AIMTo develop and evaluate a model for binary 3-year local-recurrence status among patients with ascertainable status and compare it with prespecified logistic-regression models. METHODSThis retrospective two-institution study analyzed 1035 development patients with ascertainable binary recurrence status at 3 years (331 recurrences) and 212 patients in a second-institution cohort (67 recurrences).The estimand was recurrence-status probability conditional on 3-year status being ascertainable, not cumulative incidence in all postoperative patients.The locked analytical package did not retain reason-specific exclusion counts or the source screening denominator for the second institution.Twelve predictors, 243 configurations across 14 model families, and three ensemble strategies were evaluated by five-fold outer cross-validation repeated five times with five-fold inner tuning.Mean Brier score was the primary selection criterion.The selected CatBoost model was fitted in the complete development cohort, frozen, and applied without recalibration in a protocol-frozen second-institution re-evaluation. RESULTSCatBoost had a mean internal area under the receiver operating characteristic curve (AUC) of 0.8126 and Brier score of 0.1584.In the second institution, its AUC was 0.8050 4 / 37 [95% confidence interval (CI): 0.7433-0.8668],Brier score 0.1615 (95%CI: 0.1331-0.1912),log loss 0.4884 (95%CI: 0.4146-0.5678),calibration intercept 0.148 (95%CI: -0.222 to 0.490), and calibration slope 0.985 (95%CI: 0.703-1.357).The CatBoost-minus-six-variable-logistic differences were -0.0083 for AUC (95%CI: -0.0480 to 0.0299) and 0.0043 for Brier score (95%CI: -0.0112 to 0.0198).A secondary full 12-variable logistic model also showed no material difference from CatBoost. CONCLUSIONIn the protocol-frozen second-institution re-evaluation, CatBoost retained discrimination but did not demonstrate incremental performance over logistic comparators.The study was not designed or powered to establish equivalence or non-inferiority.Prospective multicenter evaluation using time-to-event outcomes is required before clinical use.