Temporal external validation of random forest and XGBoost models for identifying concurrent diabetic retinopathy among hospitalized patients with type 2 diabetes mellitus: a single-centre temporal external validation study

Abstract Background Machine-learning prediction models for diabetic retinopathy are frequently reported with promising internal performance, yet high-quality temporal external validation remains scarce. We aimed to conduct temporal external validation of locked random forest (RF) and XGBoost models for identifying concurrent diabetic retinopathy (DR) in hospitalized patients with type 2 diabetes mellitus (T2DM). Methods This single-centre temporal external validation study used a development cohort of 3 874 hospitalized T2DM patients (2018–2025), split into training (n = 2 711) and internal test (n = 1 163) datasets. After finalising all predictors, pre-processing, hyper-parameters and classification thresholds, we validated the fixed models in 670 consecutive T2DM inpatients enrolled June-October 2025. We assessed discrimination (ROC-AUC, PR-AUC), Brier score, calibration metrics, fixed-threshold classification indices and decision-curve-analysis-derived net clinical benefit. Standardised mean differences quantified predictor distribution shifts between cohorts. paired DeLong method compared ROC-AUC within each cohort. Results In the internal test set, RF and XGBoost showed comparable discrimination (ROC-AUC 0.880 versus 0.877, P = 0.607). In temporal external validation, both models experienced performance decay; RF achieved significantly higher ROC-AUC than XGBoost (0.807 vs 0.783, P = 0.015). Only TyG-BMI exhibited substantial inter-cohort distribution shift (|standardised mean difference| = 0.419). Both models systematically underestimated DR risk but displayed heterogeneous miscalibration patterns: calibration slope 1.544 for RF and 0.668 for XGBoost. Positive net clinical benefit was observed for both models across low-to-moderate threshold probabilities. Sensitivity markedly decreased for pre-set classification thresholds in validation data. Conclusions Under temporal external validation, the locked RF model outperformed XGBoost for identifying concurrent DR in hospitalised T2DM patients. Both models underestimated disease risk and presented divergent miscalibration. Further multi-institutional external validation is required to establish real-world generalisability.