On the energy efficiency of MapReduce shuffling operations in data centers

Abstract

This paper aims to quantitatively measure the impact of different data centers networking topologies on the performance and energy efficiency of shuffling operations in MapReduce. Mixed Integer Linear Programming (MILP) models are utilized to optimize the shuffling in several data center topologies with electronic, hybrid, and all-optical switching while maximizing the throughput and reducing the power consumption. The results indicate that the networking topology has a significant impact on the performance of MapReduce. They also indicate that with comparable performance, optical-based data centers can achieve an average of 54% reduction in the energy consumption when compared to electronic switching data centers. Keywords: Data Center Networking (DCN), MapReduce, energy efficiency, completion time. INTRODUCTIONThe MapReduce programming model and its widely-used platform, Hadoop, are enabling several costeffective cloud-based big data services [1]. These services typically require extensive all-to-all communications between hosting servers leading to increased congestion and power consumption in data centers. Moreover, they result in the East-West traffic dominating over the South-North traffic. This new traffic trend has become the focus in designing state-of-art production data centers [2]. These challenges are increasingly motivating the consideration of all-optical networking in future data centers to cope with the increasing demands of big data applications while improving the data centers performance and decreasing their power consumption [3].The processing in MapReduce is composed of map, shuffle, and reduce phases. The input data is stored in several servers' local disks and is globally managed by a distributed file system (DFS) [1]. The processing starts by assigning map slots according to the number of input data chunks and available computing resources, and reduce slots according to the user's configurations. If chunks are more than map slots, the map phase will run in several waves according to their scheduling [4]. Each map slot processes it assigned chunks, preferably available locally, and generates intermediate results in the form of < key,value> pairs. The intermediate results are shuffled to reduce slots according to their keys where each slot is assigned to process a unique set of keys [1]. Finally, each reduce slot sorts its inputs, calculates final results, and saves them in the DFS.Several optimization studies have been carried out by both academia and industry to enhance the performance and energy efficiency of big data applications (e.g. [2], [4]- [21]). The performance of big data applications and frameworks such as MapReduce is associated with a wide range of factors and parameters such as the cluster specifications (e.g. CPU, memory, networking, and disk I/O resources [9]), framework used or version, in addition to selected configurations and mechanisms for data and jobs placements and tasks scheduling [4]- [8]. Moreover, as the deployments of big data applications are evolvin...

Impact of Link Failures on the Performance of MapReduce in Data Center Networks

“…Indy GraySort benchmark [20] is selected as it is a representative workload for examining the congestion in DCNs due to routing intermediate data that is equal in size to the input data from the 10 map workers to the 6 reduce workers. The power consumption values of the selected electronic and optical equipment are as in [12], and [15]. Figure 2 provides the shuffling completion time results for a maximum server rate of 1000 Mbytes/s and input data ranging from 1 to 20 GBytes in different DCNs.…”

“…Figure 2 provides the shuffling completion time results for a maximum server rate of 1000 Mbytes/s and input data ranging from 1 to 20 GBytes in different DCNs. The subfigures, show the results in [12], and [13] for the cases where all links are working, in addition to the completion time results under 2 or 3 cases of links failures in each DCN. For the Spine-and-Leaf architecture, the degradation due to links 1-3 disconnection is less than 2-6 because the latter is more utilized due to serving the flows to 3 reduce workers compared to 1 reduce worker by links 1-3.…”

Performance Model of MapReduce Iterative Applications for Hybrid Cloud Bursting

“…To this end, we choose a set of representative shuffle sizes, measure the shuffle overhead per reducer using the synthetic benchmark, and then apply linear regression to obtain two approximation functions: one for the average (using all reducers) and one for the maximum (using only the slowest reducer for each shuffle size). Note that we have chosen linear regression because it was confirmed by previous work as a good approximation for shuffle behavior [22], [33], [34] for a variety of network topologies with different performance characteristics. In our case, we assume that the weak link will slow data transfers uniformly, thus preserving the linear behavior.…”

Energy Efficiency of Server-Centric PON Data Center Architecture for Fog Computing

“…As with cloud networking infrastructures, the increasing demands of data processing are also challenging the DCN because most of big data processing applications require extensive all-to-all server communications due to their distributed nature [14]. The impact of state-of-the-art DCN topologies on the performance and/or the energy efficiency of big data applications have been considered in [15]- [18]. To overcome the increasing power consumption and congestions in current data centers, and to meet the heterogeneous performance requirements of big data applications, an increasing number of hybrid and all-optical DCNs are proposed.…”

“…In this paper, we compare the energy consumption and completion time for MapReduce sort workloads as in our previous work in [18] while additionally considering the server-centric PON-based DCN in [24]. The rest of this paper is organized as the following.…”