Research on Optimization Algorithms for High-Performance Computing on Large-Scale Data Sets

This paper proposes an efficient parallel adaptive neighborhood search optimization algorithm (PASNSO) to solve the optimization problem on large-scale data sets. The algorithm combines the advantages of local search and global search and introduces an adaptive mechanism to achieve the goal of dynamically adjusting the search strategy at different stages. In order to further improve the computational efficiency, this paper parallelizes the PASNSO algorithm and implements distributed computing with the help of the MPI framework. This parallel processing method not only improves the computing speed, but also makes full use of multi-core and cluster computing resources, especially when processing million-dimensional data sets, it can greatly shorten the computing time. Experimental results show that PASNSO performs significantly better than traditional optimization algorithms such as PSO and GA on multiple large-scale data sets. In the experiment on a 1 million-dimensional data set, PASNSO improves the convergence speed by about 30% and the computational accuracy by 15%. These experimental results prove that PASNSO not only has obvious advantages in optimization accuracy, but also can process larger-scale data sets in a high-performance computing environment, and has a wide range of application potential.