A Data Generator for Cloud-Scale Benchmarking

Rabl, Tilmann; Frank, Michael; Sergieh, Hatem Mousselly; Kosch, Harald

doi:10.1007/978-3-642-18206-8_4

Cited by 61 publications

(42 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, Cloud elasticity is related not only to the resource scaling time but also to the resource charging basis (Islam et al, 2012); consequently, it has become a challenge to explicitly quantify the amount of elasticity of a Cloud service (Li et al, 2012b(Li et al, , 2013c. Therefore, to cover and test various service aspects from a holistic view, the current practitioners normally suggest employing benchmark suites for Cloud services evaluation (Iosup et al, 2011;Rabl et al, 2010). For example, the kernel benchmarks in NPB have been used to reveal different micro features of Amazon EC2 like computation, communication and storage respectively (Akioka and Muraoka, 2010); while six scale-out workloads are collected to simulate different macro application scenarios in today's Cloud infrastructure (Ferdman et al, 2012).…”

Section: Related Workmentioning

confidence: 99%

“…In fact, according to the rich research in the evaluation of traditional computer systems, the selection of metrics plays an essential role in evaluation implementations (Li et al, 2012b). Particularly, it is often useful and significant to evaluate Cloud services from a holistic view (Iosup et al, 2011;Rabl et al, 2010), and using single measurement indexwould be helpful and convenient for comparing alternatives and drawing conclusions (Islam et al, 2012). More importantly, a single index of an overall measurement can play a summary Response role in experimental design and analysis (Montgomery, 2009) for evaluating Cloud services.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Boosting Metrics: Measuring Cloud Services From the Holistic Perspective

Li¹,

O’Brien²,

Zhang³

2014

STCC

View full text Add to dashboard Cite

Studies have shown that Cloud services evaluation would be crucial and beneficial for both service customers and providers, and metrics would play a vital role in any evaluation implementation. Considering the numerous and various aspects of Cloud services, a frequent suggestion is to perform evaluation from a holistic view. The currently normal strategy of holistic evaluation is to use a set of metrics along with a suite of benchmarks to conduct separated experiments. Given the separated, diverse, and even possibly conflicting measurement criteria, it could be still hard for customers with such evaluation reports to understand an evaluated Cloud service from a global perspective. Inspired by the boosting approaches to machine learning, we proposed the concept Boosting Metrics to represent all the potential approaches that are able to deliver summary measurement of Cloud services. Essentially, the idea of boosting metrics is to holistically measure Cloud services with concern of service properties, which supplements the strategy of employing benchmark suites that is to holistically evaluate Cloud services with concern of different workloads. This paper introduces two types of preliminary approaches, and unifies a set of sophisticated measurements into the notion of boosting metrics. In particular, we show that boosting metrics can be used as a summary Response for applying experimental design to Cloud services evaluation. Although the concept Boosting Metrics was refined based on our work in the Cloud Computing domain, we believe it can be easily adapted to the evaluation work of other computing paradigms.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Boosting Metrics: Measuring Cloud Services From the Holistic Perspective

Li¹,

O’Brien²,

Zhang³

2014

STCC

View full text Add to dashboard Cite

show abstract

“…Detailed descriptions of the architecture and functionality can be found in previous publications [15,6]. PDGF is a generic data generation framework that was built around the principle of parallel pseudo random number generation.…”

Section: Parallel Data Generation Frameworkmentioning

confidence: 99%

“…One of our first non-trivial data sets was the TPC-H data set which we implemented to compare the performance of PDGF to DBGen [15]. TPC-H is a data warehousing benchmark that is widely used in industry and academia.…”

Section: Tpc-h and Ssbmentioning

confidence: 99%

See 1 more Smart Citation

Rapid development of data generators using meta generators in PDGF

Rabl

Poess

Danisch

et al. 2013

Proceedings of the Sixth International Workshop on Testing Database Systems

Self Cite

View full text Add to dashboard Cite

Generating data sets for the performance testing of database systems on a particular hardware configuration and application domain is a very time consuming and tedious process. It is time consuming, because of the large amount of data that needs to be generated and tedious, because new data generators might need to be developed or existing once adjusted. The difficulty in generating this data is amplified by constant advances in hardware and software that allow the testing of ever larger and more complicated systems. In this paper, we present an approach for rapidly developing customized data generators. Our approach, which is based on the Parallel Data Generator Framework (PDGF), deploys a new concept of so called meta generators. Meta generators extend the concept of column-based generators in PDGF. Deploying meta generators in PDGF significantly reduces the development effort of customized data generators, it facilitates their debugging and eases their maintenance.

show abstract

The impact of columnar file formats on SQL‐on‐hadoop engine performance: A study on ORC and Parquet

Ivanov

Pergolesi

2019

Concurrency and Computation

View full text Add to dashboard Cite

Columnar file formats provide an efficient way to store data to be queried by SQL-on-Hadoop engines. Related works consider the performance of processing engine and file format together, which makes it impossible to predict their individual impact. In this work, we propose an alternative approach: by executing each file format on the same processing engine, we compare the different file formats as well as their different parameter settings. We apply our strategy to two processing engines, Hive and SparkSQL, and evaluate the performance of two columnar file formats, ORC and Parquet. We use BigBench (TPCx-BB), a standardized application-level benchmark for Big Data scenarios. Our experiments confirm that the file format selection and its configuration significantly affect the overall performance. We show that ORC generally performs better on Hive, whereas Parquet achieves best performance with SparkSQL. Using ZLIB compression brings up to 60.2% improvement with ORC, while Parquet achieves up to 7% improvement with Snappy. Exceptions are the queries involving text processing, which do not benefit from using any compression.

show abstract

A Data Generator for Cloud-Scale Benchmarking

Cited by 61 publications

References 16 publications

Boosting Metrics: Measuring Cloud Services From the Holistic Perspective

Boosting Metrics: Measuring Cloud Services From the Holistic Perspective

Rapid development of data generators using meta generators in PDGF

The impact of columnar file formats on SQL‐on‐hadoop engine performance: A study on ORC and Parquet

Contact Info

Product

Resources

About