Automation and parallelization scheme to accelerate pulsar observation data processing

Previous research focuses on enhancement algorithms, using the GPU to speed up programs, and thread-level parallelism. These methods overlook maximizing the utilization of existing CPU resources and reducing human and time costs through process automation. To address this, this paper proposes a scheme that combines “Srun job submission mode”, “Sbatch job submission mode and “Monitor function” (SSM). The scheme includes three main modules: data management, command management, and resource management. Its core innovations are command splitting and parallel execution. The results show that this method effectively improves CPU utilization and reduces the time required for data processing. In terms of CPU utilization, the average value of this scheme is 89%. In contrast, the average CPU utilization of “Srun job submission mode” and “Sbatch job submission mode” is significantly lower, which is 43% and 52%. In terms of data processing time, SSM testing on the FAST’s data requires only five and a half hours, compared to 8 hours in the “Srun job submission mode” and 14 hours in the “Sbatch job submission mode”. In addition, tests on FAST and Parkes datasets demonstrate the universality of the SSM scheme, which can process data from different telescopes. The compatibility of the SSM scheme in the pulsar search is verified using two days of observational data from the globular cluster M2, successfully discovering all published pulsars in M2.