Explainable Ensemble Boosting Framework for Chronic Disease Risk Prediction Using Lifestyle and Clinical Data: A Multi-Dataset Study

Chronic diseases represent a substantial public health burden, motivating the development of transparent and reproducible approaches for disease-specific risk prediction using routinely available lifestyle and clinical variables. This study presents a common explainable ensemble boosting framework for comparative prediction across three independent disease-specific datasets: heart disease, lung cancer, and diabetes. Five boosting algorithms, including XGBoost, CatBoost, LightGBM, AdaBoost, and Gradient Boosting, were evaluated under a consistent leakage-free protocol. Each dataset was first split into stratified 80% training and 20% held-out test sets, with preprocessing, random oversampling, hyperparameter optimization, probability calibration, and threshold selection performed only on the training data. Performance was assessed using accuracy, balanced accuracy, precision, sensitivity, specificity, F1-score, ROC-AUC, PR-AUC, Brier score, and bootstrap confidence intervals. On the held-out test sets, AdaBoost achieved the highest balanced accuracy (68.07%) and ROC-AUC (71.71%) for heart disease; CatBoost achieved the highest balanced accuracy (92.59%) and ROC-AUC (96.30%) for lung cancer, although these estimates require caution because the lung-cancer test set contained only eight negative observations; and XGBoost achieved the highest balanced accuracy (77.02%), whereas Gradient Boosting achieved the highest ROC-AUC (82.43%) for diabetes. Repeated stratified cross-validation showed significant differences among the five algorithms for balanced accuracy (Friedman x2 = 26.293, p < 0.001), ROC-AUC (x2 = 21.173, p < 0.001), and F1-score (x2 = 15.093, p = 0.0045). Across these endpoints, Gradient Boosting achieved the best mean rank (1.98), followed by CatBoost (2.47), although no model was uniformly the highest-performing across all datasets and evaluation measures. SHAP and LIME analyses were additionally used to characterize global and local feature contributions to model predictions. The findings demonstrate the feasibility of a common explainable boosting workflow across heterogeneous disease-specific benchmark datasets while emphasizing that these results constitute internal benchmark evidence and require independent external and prospective validation before clinical application.