Application of K-means supported by clustered systems in big data association rule mining

Association rule mining plays an important role in the field of data mining, which is used to discover hidden relationships. However, as data volumes increase, traditional association rule mining methods are constrained to single-machine computing when processing large-scale data. These methods are unable to leverage the advantages of modern distributed computing frameworks, resulting in more significant performance bottlenecks when processing large-scale datasets. Therefore, research on how to combine distributed computing technology with association rule mining has become the key to improving efficiency and scalability. To this end, the study introduced a parallel frequent itemset mining technique, FiDoop DP, which used the MapReduce programming paradigm for data partitioning on Hadoop clusters and integrates an improved k-means++ algorithm for data preprocessing to provide better data processing results. The findings indicated that the enhanced k-means++ clustering method achieved a Davies-Bouldin index of 0.642 for performance validation, while its Calinski-Harabasz score reached 5186. The improved k-means++ clustering technique showed advantageous clustering results, while the data partitioning method based on frequent item set parallel mining shown a notable performance advantage. With 60 seed points, the execution time for the frequent item set parallel mining technique was just 683 s, the mining duration was only 402 s, and the shuffling expenditure amounted to 2280GB. This indicates that the FiDoop DP method proposed by the study has significant importance in modern cluster environments. By combining the distributed computing capabilities of Hadoop clusters with the improved k-means++ clustering algorithm, this method effectively solves the scalability problem in processing large datasets and significantly improves the efficiency of clustering analysis and frequent itemset mining.