Expert-defined features remain competitive with foundation models in predicting Crohn’s-like disease of the pouch

Abstract Patients with ulcerative colitis (UC) who undergo ileal pouch–anal anastomosis (IPAA) remain at risk of Crohn’s-like disease of the pouch (CLDP), a major cause of pouch failure. Accurate preoperative risk stratification is challenging. We evaluated whether machine learning (ML) and electronic health record–based foundation models (EHR-FMs) could improve prediction of CLDP. We included UC patients who underwent IPAA at Mount Sinai Hospital (2007–2021) with at least one postoperative pouchoscopy. Two feature sets were tested: (1) expert-defined, chart-reviewed variables and (2) embeddings generated by Mamba FM on EHR data from Mount Sinai Data Warehouse (MSDW), which were pretrained on external data (Stanford Medicine), local data (MSDW), or hybrid fine-tuning (Stanford-MSDW). Features were modeled with eXtreme Gradient Boosting (XGBoost), performance assessed via nested cross-validation, and interpretability evaluated with SHapley Additive exPlanations (SHAP). CLDP occurred in 80/447 patients (17.9%) within a median of 2.97 [1.82; 5.43] years after surgery. The FM trained only on Stanford data performed poorly on our data (perplexity = 39.32), but local retraining improved performance (MSDW: 2.45), with further gains from hybrid fine-tuning (Stanford-MSDW: 2.13). Stanford-MSDW embeddings achieved the best FM-based results (macro F1 = 0.55, AUROC = 0.62). Chart-reviewed features showed numerically higher performance (macro F1 = 0.58, AUROC = 0.65). However, the difference was not statistically significant (F1 Score = 0.03, 95% CI −0.04, 0.11, p-value = 0.41). SHAP analyses highlighted age at surgery, BMI, disease duration, and colectomy severity as key predictors. To date, expert-defined, chart-reviewed features predicted CLDP comparably to current EHR-FM embeddings and remain the more interpretable option.