Design and Implementation of Parallel Processing Algorithm for Big Data in High Performance Computing Framework

This paper discusses the design and implementation of parallel processing algorithm for big data under the framework of high performance computing (HPC). In the design of parallel processing algorithm for big data, this paper proposes a partition method based on data characteristics, which uses K-means clustering algorithm to divide the data set into several subsets, with high correlation within each subset and low correlation among subsets, so as to improve the efficiency of parallel processing. The improved MapReduce model is adopted as the parallel processing framework, and the data processing flow is further optimized by introducing preprocessing steps to reduce the transmission of invalid data and redundant data. In addition, this paper also realizes the dynamic load balancing and fault tolerance mechanism, and ensures the efficiency and stability of the algorithm through dynamic task allocation, redundant data and calculation strategy. Experimental results show that the proposed algorithm performs well on 10TB BigDataBench dataset and 5TB ProductionDataset dataset. Compared with the traditional MapReduce algorithm, the processing time of the new algorithm is reduced by about 30% on BigDataBench dataset and by about 25% on ProductionDataset dataset. At the same time, in terms of resource utilization, the memory utilization and CPU utilization of the new algorithm are also significantly higher than those of the traditional algorithm, showing higher comprehensive performance. The research results of this paper provide new ideas and methods for the design and optimization of big data parallel processing algorithms, and are of great significance for promoting the development of big data processing technology under HPC framework.