DK-MS: an efficient method for solving imbalanced Big Data classification problems

In recent years, Big Data has emerged as a crucial source for data mining, encompassing a vast and complex collection of structured and unstructured data. Machine learning has become widely adopted for analyzing this data and deriving structured insights, particularly for Big Data Mining classification. To fully utilize this valuable resource, new tools and learning methods are needed to address scalability challenges, limited computation time, and storage capacity. Big Data processing and management require data-driven algorithms and statistical models, which help analyze datasets, identify patterns, and make predictions. However, class imbalance is a common challenge in Big Data mining. This paper introduces a new method called "DK-MS" to address imbalanced Big Data classification problems. DK-MS, based on Double K-Means and SMOTE, aims to reduce the volume of big datasets while preserving essential characteristics and ensuring information reliability. By employing classifiers like Logistic Regression, K-NN, Naive Bayes, and Random Forests, the DK-MS method achieves higher accuracy rates and AUC measures compared to cases without data balancing strategies. The DK-MS method demonstrated high accuracy rates of 91.30%, 99.93%, and 99.93%, demonstrating its significant contribution to effectively addressing imbalanced Big Data classification problems.