I/O Characterization of Big Data Workloads in Data Centers

Pan, Fengfeng; Yue, Yinliang; Xiong, Jin; Hao, Daxiang

doi:10.1007/978-3-319-13021-7_7

Cited by 21 publications

(8 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We compare our work with previous studies on real-world, largescale storage workloads, including big data. Overall, because of our focus on object stores, our work complements the body of work done on hardware-level storage for big data workloads, e.g., [21].…”

Section: Related Workmentioning

confidence: 97%

Characterization of a Big Data Storage Workload in the Cloud

Talluri

Łuszczak²,

Abad

et al. 2019

Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering

View full text Add to dashboard Cite

The proliferation of big data processing platforms has led to radically different system designs, such as MapReduce and the newer Spark. Understanding the workloads of such systems facilitates tuning and could foster new designs. However, whereas MapReduce workloads have been characterized extensively, relatively little public knowledge exists about the characteristics of Spark workloads in representative environments. To address this problem, in this work we collect and analyze a 6-month Spark workload from a major provider of big data processing services, Databricks. Our analysis focuses on a number of key features, such as the long-term trends of reads and modifications, the statistical properties of reads, and the popularity of clusters and of file formats. Overall, we present numerous findings that could form the basis of new systems studies and designs. Our quantitative evidence and its analysis suggest the existence of daily and weekly load imbalances, of heavy-tailed and bursty behaviour, of the relative rarity of modifications, and of proliferation of big data specific formats.

show abstract

Section: Related Workmentioning

confidence: 97%

Characterization of a Big Data Storage Workload in the Cloud

Talluri

Łuszczak²,

Abad

et al. 2019

Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering

View full text Add to dashboard Cite

show abstract

“…e I/O workload is read-intensive and is performed on data residing on the node locally. K-means clustering is typically I/O bound [53].…”

Section: Applications' I/o Requirementsmentioning

confidence: 99%

Labios

Kougkas

Devarajan

Lofstead

et al. 2019

Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing

View full text Add to dashboard Cite

In the era of data-intensive computing, large-scale applications, in both scienti c and the BigData communities, demonstrate unique I/O requirements leading to a proliferation of di erent storage devices and so ware stacks, many of which have con icting requirements. In this paper, we investigate how to support a wide variety of con icting I/O workloads under a single storage system. We introduce the idea of a Label, a new data representation, and, we present LABIOS: a new, distributed, Label-based I/O system. LABIOS boosts I/O performance by up to 17x via asynchronous I/O, supports heterogeneous storage resources, o ers storage elasticity, and promotes in-situ analytics via data provisioning. LABIOS demonstrates the e ectiveness of storage bridging to support the convergence of HPC and BigData workloads on a single platform. CCS CONCEPTS •Information systems → Distributed storage; Hierarchical storage management; Storage power management; •Computer systems organization → Distributed architectures; Data ow architectures; Heterogeneous (hybrid) systems;

show abstract

“…Many Big Data computations are I/O-bound and thus may benefit from faster hardware at bottlenecks [69]. For datasets that are distributed over the whole cluster, performance can be improved by using solid state disks (SSDs) [44,81] and faster network interconnects such as 10 gigabit Ethernet and Infiniband [81].…”

Section: Using Faster Hardwarementioning

confidence: 99%

A Distributed Stream Library for Java 8

Chan

Wellings

Gray

et al. 2017

IEEE Trans. Big Data

View full text Add to dashboard Cite

An increasingly popular application of parallel computing is Big Data, which concerns the storage and analysis of very large datasets. Many of the prominent Big Data frameworks are written in Java or JVM-based languages. However, as a base language for Big Data systems, Java still lacks a number of important capabilities such as processing very large datasets and distributing the computation over multiple machines. The introduction of Streams in Java 8 has provided a useful programming model for data-parallel computing, but it is limited to a single JVM and still does not address Big Data issues.This thesis contends that though the Java 8 Stream framework is inadequate to support the development of Big Data applications, it is possible to extend the framework to achieve performance comparable to or exceeding those of popular Big Data frameworks. It first

show abstract

I/O Characterization of Big Data Workloads in Data Centers

Cited by 21 publications

References 15 publications

Characterization of a Big Data Storage Workload in the Cloud

Characterization of a Big Data Storage Workload in the Cloud

Labios

A Distributed Stream Library for Java 8

Contact Info

Product

Resources

About