Research on Distributed Load Simulation Methods for Big Data Oriented Towards Resource Elastic Management

This article presents a distributed load simulation method for resource elastic management. Through this method, the load simulation tools can be deployed quickly and the parallelization granularity of test tasks can be controlled flexibly. Hadoop runtime system is responsible for managing intermediate files and failed test tasks during the test process, which has good fault tolerance. The load simulation results are stored in the distributed file system for data mining again. The main contributions of this article are as follows: (1) A task scheduling and task synchronization strategy for distributed load simulation is proposed, which can realize more accurate load simulation and is conducive to the reproduction of real load scenarios. (2) A partition algorithm for load simulation data is proposed, which can effectively reduce the imbalance of test data among different tasks and shorten the evaluation time. (3) The strategy of synchronization and data recovery balance for distributed load simulation tasks is proposed and verified by experiments. In the distributed load simulation environment, the effectiveness of the proposed parallelization method is verified.