High Performance Cloud Ready Parallel Batch Processing using Auto Scaling Containers and Queue Technique

In order to solve the problems of the current batch processing systems as well as to reduce cost and shorten processing times, an enhanced parallel processing strategy has been introduced. Existing Batch system are running in dedicated servers and at most one Job can be processed at a time. Which results in higher costs. Also, batch jobs run in sequential to avoid duplicate processing of data so it will take more time. Where as in our proposed system we use queue to avoid duplicate and containers to achieve parallel processing and once job completed containers will be decommissioned and no need to pay for cloud provider when not in use. Hence, this proposed system is using queue mechanisms like RabbitMQ and Multiple containers(pods) all the containers can run parallelly and get inputs from queue so that duplicate processing will not happen in parallel processing to complete the job faster. This is achieved by Auto scaling mechanism, and the paper proposes a novel algorithm called Chunk Optimized Processing based on Queue Length (COPQL). Particularly, one of the main parts of this system is the Initiator, whose job it is to read data from source and push messages(data) into the queue as chunks, especially the primary keys from the database. RabbitMQ queues has been utilized by the proposed system, which are configured with specific topics or queues for each job type and set up as a 3-node cluster to ensure fault tolerance. Multiple processors (containerized nodes) have been included in the system for each job type, which can automatically scale based on the new Chunk Optimized Processing based on Queue Length (COPQL) algorithm. Each processor reads a chunk, consisting of a group of primary keys, retrieves the corresponding data from the source database, performs the necessary business processing, and updates the status in the Batch DB upon completion. It is highly notable that other features include a retry mechanism for handling failures in specific chunks, a caching mechanism to improve processing speed, and a tracking mechanism to monitor the batch process and to generate final reports. Importantly, the COPQL algorithm has been designed to achieve Low cost (serverless model) and minimal processing time for batch. Particularly, the system has been implemented by using Python, and the results have been analyzed based on weighted sum completion time and the effect of the number of tasks per user.