Aggregation-based partitioning algorithm for traffic congestion in MapReduce

The current era has witnessed a remarkable transformation in scientific frontiers, largely driven by advancements in the digital domain. This has resulted in an unprecedented explosion of data known as big data. Among the platforms capable of effectively handling massive data volumes cost-effectively, MapReduce stands out. While previous research has focused on enhancing MapReduce’s overall performance by selecting and scheduling mappers, little focus has been given to optimizing the shuffle phase’s impact on performance. MapReduce operates through multiple phases, with the shuffle phase generating substantial data traffic in the case of heavy jobs. Optimizing or encapsulating aspects, e.g., catalyst, can significantly accelerate the platform’s performance. This work introduces an aggregation-based partitioning algorithm (ABPA) that addresses the limitations of existing approaches commonly adopted to reduce traffic congestion during the intermediate, i.e., shuffle, phase of MapReduce. The proposed ABPA algorithm is evaluated through several experiments involving different data sets of varying types and lengths, employing different numbers of partitions, each containing a specified number of mappers and an aggregator. The experimental results demonstrated a significant reduction in network traffic costs and improved total MapReduce job execution time when using the ABPA scheme. Specifically, the proposed algorithm achieved 73% and 56% network traffic cost improvement over basic hash and conventional aggregation schemes, respectively. Additionally, it reduced total job completion time by 60% and 46% compared to the same schemes.