Workload characterization on a production Hadoop cluster: A case study on Taobao

Abstract

Abstract-MapReduce is becoming the state-of-the-art computing paradigm for processing large-scale datasets on a large cluster with tens or thousands of nodes. It has been widely used in various fields such as e-commerce, Web search, social networks, and scientific computation. Understanding the characteristics of MapReduce workloads is the key to achieving better configuration decisions and improving the system throughput. However, workload characterization of MapReduce, especially in a largescale production environment, has not been well studied yet.To gain insight on MapReduce workloads, we collected a two-week workload trace from a 2,000-node Hadoop cluster at Taobao, which is the biggest online e-commerce enterprise in Asia, ranked 14 ℎ in the world as reported by Alexa. The workload trace covered 912,157 jobs, logged from Dec. 4 to Dec. 20, 2011. We characterized the workload at the granularity of job and task, respectively and concluded with a set of interesting observations. The results of workload characterization are representative and generally consistent with data platforms for e-commerce websites, which can help other researchers and engineers understand the performance and job characteristics of Hadoop in their production environments. In addition, we use these job analysis statistics to derive several implications for potential performance optimization solutions.

Tempo

“…For example, we developed a statistical model based on one month of historical traces from Company ABC's production database workload. The workload distributions from Company ABC (reported further in the evaluation section) are similar to the distributions described in [40]. In particular, the task duration approximately follows a lognormal distribution, and the job arrival approximately follows a Poisson process.…”

A Pareto-based scheduler for exploring cost-performance trade-offs for MapReduce workloads

“…For the simulation workloads, we use traces from Yahoo! [28] and Taobao [29], while for the empirical workloads, we execute short-running jobs from the PUMA benchmark [30] to display the applicability of our framework in commonly used workloads from major companies like Twitter. Our experimental results illustrate that our search algorithm outperforms state-of-the-art schemes by a few orders of magnitude in terms of execution time.…”

Rethinking Design Metrics for Datacenter DRAM

“…Hence, as the number of users increases, the role of the in-memory caching tier becomes increasingly important to meet the service level agreements/requirements (SLAs/SLRs) for the services under consideration. Similar to KV stores, applications at other tiers of the datacenter also require access to DRAM, although, after a certain point, the applications are not sensitive to DRAM [3,4,5]. Some of our previous work in this regard [3] characterized the most popular application for each server tier.…”