A Scalable Framework for Precision-Boosted Healthcare Predictions with Advanced Algorithms

Healthcare data continues to grow in size and complexity. Hence, there's a rising need for more scalable and accurate prediction models. This study presents a framework that combines big data processing with advanced machine learning techniques to improve healthcare predictions, with diabetes mellitus as the primary use case. Apache Spark is used for data preprocessing, which distributes the large dataset to its worker nodes and completes the tasks at faster rate. This approach significantly reduces processing time when handling with millions of records. Most of the healthcare data often suffers from class imbalance, which is addressed with Synthetic Minority Over-sampling Technique (SMOTE) to develop a well-learned model. A StackingClassifier is built with three powerful boosting algorithms: XGBoost, CatBoost, and LightGBM. These models are combined to capture the strength of each algorithm and then enhanced by an AdaBoost meta-learner to boost precision. Further, doing cross validation and tuning the model with hyperparameters showed impressive results. SHAP and LIME is used for explaining the model's predictions. Evaluating the model with standard metrics like accuracy, precision, recall, and AUC-ROC shows up 98% accuracy and precision. This high precision indicates fewer false positives, making the model especially useful for real-world healthcare applications.