Characterizing Network Traffic in a Cluster-based, Multi-tier Data Center

Ersoz, Deniz; Yousif, Mazin; Das, Chita R.

doi:10.1109/icdcs.2007.90

Cited by 97 publications

(63 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In [9], Ersoz et al measure the network traffic characteristics in a cluster-based, multi-tier data center. This study is not based on a real commercial data center service.…”

Section: Related Workmentioning

confidence: 99%

The Impact of Virtualization on Network Performance of Amazon EC2 Data Center

Wang

2010

2010 Proceedings IEEE INFOCOM

509

289

View full text Add to dashboard Cite

Abstract-Cloud computing services allow users to lease computing resources from large scale data centers operated by service providers. Using cloud services, users can deploy a wide variety of applications dynamically and on-demand. Most cloud service providers use machine virtualization to provide flexible and costeffective resource sharing. However, few studies have investigated the impact of machine virtualization in the cloud on networking performance.In this paper, we present a measurement study to characterize the impact of virtualization on the networking performance of the Amazon Elastic Cloud Computing (EC2) data center. We measure the processor sharing, packet delay, TCP/UDP throughput and packet loss among Amazon EC2 virtual machines. Our results show that even though the data center network is lightly utilized, virtualization can still cause significant throughput instability and abnormal delay variations. We discuss the implications of our findings on several classes of applications.

show abstract

“…In [9], Ersoz et al measure the network traffic characteristics in a cluster-based, multi-tier data center. This study is not based on a real commercial data center service.…”

Section: Related Workmentioning

confidence: 99%

The Impact of Virtualization on Network Performance of Amazon EC2 Data Center

Wang

2010

2010 Proceedings IEEE INFOCOM

509

289

View full text Add to dashboard Cite

show abstract

“…Many previous efforts have been accomplished in different areas, including network system [18], [19], storage system [20], [21], [22], Web server [23], [24], and HPC cluster [25]. Both network and storage subsystems are key components for the Hadoop system.…”

Section: Related Workmentioning

confidence: 99%

Workload characterization on a production Hadoop cluster: A case study on Taobao

Ren

Wan

et al. 2012

2012 IEEE International Symposium on Workload Characterization (IISWC)

View full text Add to dashboard Cite

Abstract-MapReduce is becoming the state-of-the-art computing paradigm for processing large-scale datasets on a large cluster with tens or thousands of nodes. It has been widely used in various fields such as e-commerce, Web search, social networks, and scientific computation. Understanding the characteristics of MapReduce workloads is the key to achieving better configuration decisions and improving the system throughput. However, workload characterization of MapReduce, especially in a largescale production environment, has not been well studied yet.To gain insight on MapReduce workloads, we collected a two-week workload trace from a 2,000-node Hadoop cluster at Taobao, which is the biggest online e-commerce enterprise in Asia, ranked 14 ℎ in the world as reported by Alexa. The workload trace covered 912,157 jobs, logged from Dec. 4 to Dec. 20, 2011. We characterized the workload at the granularity of job and task, respectively and concluded with a set of interesting observations. The results of workload characterization are representative and generally consistent with data platforms for e-commerce websites, which can help other researchers and engineers understand the performance and job characteristics of Hadoop in their production environments. In addition, we use these job analysis statistics to derive several implications for potential performance optimization solutions.

show abstract

“…They design a generic, app-independent profiler that monitors TPC at the socket-level and identifies performance bottlenecks within the same and across different connections. Ersoz et al [11] also characterize the network traffic of a multi-tier DC. They observe that inter-arrival times and message sizes follow log-normal distributions, while service times fall within the Pareto distribution and show heavy tails at heavy loads.…”

Section: Related Workmentioning

confidence: 99%

ECHO: Recreating network traffic maps for datacenters with tens of thousands of servers

Delimitrou

Sankar

Kansal

et al. 2012

2012 IEEE International Symposium on Workload Characterization (IISWC)

View full text Add to dashboard Cite

Abstract-Large-scale datacenters now host a large part of the world's data and computation, which makes their design a crucial architectural challenge. Datacenter (DC) applications, unlike traditional workloads, are dominated by user patterns that only emerge in the large-scale. This creates the need for concise, accurate and scalable analytical models that capture both their temporal and spatial features and can be used to create representative activity patterns. Unfortunately, previous work lacks the ability to track the complex patterns that are present in these applications, or scales poorly with the size of the system. In this work, we focus on the network aspect of datacenter workloads. We present ECHO, a scalable and accurate modeling scheme that uses hierarchical Markov Chains to capture the network activity of large-scale applications in time and space. ECHO can also use these models to re-create representative network traffic patterns. We validate the model against real DC-scale applications, such as Websearch and show marginal deviations between original and generated workloads. We verify that ECHO captures all the critical features of DC workloads, such as the locality of communication and burstiness and evaluate the granularity necessary for this. Finally we perform a detailed characterization of the network traffic for workloads in DCs of tens of thousands of servers over significant time frames.

show abstract

Characterizing Network Traffic in a Cluster-based, Multi-tier Data Center

Abstract: With the increasing use of various Web-based

Cited by 97 publications

References 24 publications

The Impact of Virtualization on Network Performance of Amazon EC2 Data Center

The Impact of Virtualization on Network Performance of Amazon EC2 Data Center

Workload characterization on a production Hadoop cluster: A case study on Taobao

ECHO: Recreating network traffic maps for datacenters with tens of thousands of servers

Contact Info

Product

Resources

About