Early prediction of diabetes using a reduced and interpretable feature set with a scalable machine learning framework

Abstract Early detection of diabetes is crucial for effective intervention and management. This study presents a scalable and interpretable machine learning framework using a real-world dataset obtained from Kaggle consisting of 96,146 valid records after removal of duplicate entries. The dataset contains clinically relevant and demographically diverse features, including age, gender, BMI, hypertension, heart disease, smoking history, HbA1c level, and blood glucose level. The framework includes enhanced preprocessing and advanced class balancing using Synthetic Minority Oversampling Technique (SMOTE). Two evaluation approaches, one using the full feature set and another using a reduced subset of top features identified through feature importance analysis and Recursive Feature Elimination (RFE) with Random Forest (RF) as the base estimator. In the full-feature pipeline, classifiers including Logistic Regression (LR), Decision Tree (DT), RF, and XGBoost are trained and evaluated. The RF model is fine-tuned using a randomized search, and the performance of the XGBoost model is improved by using the grid search. Model interpretability is assessed using LIME analysis, which asserts the role of smoking history and hypertension on the full and reduced feature set using RF. The tuned XGBoost model achieves the highest accuracy of 96.8% and a ROC-AUC of 0.973. In the reduced-feature pipeline, an RF trained on the top five features achieves an ROC-AUC of 0.958 with a lower computational cost. The results demonstrate that high predictive accuracy can be maintained with a minimal and interpretable feature set, making the proposed framework suitable for deployment in real-world, resource-constrained environments.