Bandwidth-Aware Routing Mechanism to Control Hadoop Shuffle Traffic over SoftwareDefined Networking

MapReduce is the main program in the Hadoop computing architecture.It involves mapping and reduction processes that require servers to exchange large amounts of data with each other.Data exchange between servers during the shuffle stage can lead to insufficient network bandwidth.In the present study, an algorithm for distributing Hadoop shuffle traffic to all possible paths was developed.Software-defined networking (SDN), a centrally controlled form of network architecture, was used to collect network status parameters to allocate shuffle traffic.During traffic distribution, the Mininet simulator was used to build the network topology and simulate operation.The Ryu controller was used as the SDN controller.The simulation results indicated that the proposed method was superior to the Spanning Tree Protocol (STP) and bandwidth-aware algorithm in reducing Hadoop completion time through effective shuffle traffic distribution.