BDGS: A Scalable Big Data Generator Suite in Big Data Benchmarking

Ming, Zijian; Luo, Chunjie; Gao, Wanling; Han, Rui; Yang, Qiang; Wang, Lei; Zhan, Jianfeng

doi:10.1007/978-3-319-10596-3_11

Cited by 68 publications

(50 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Spark SQL queries from BigDataBench have been reprogrammed to use DataFrame API. Big Data Generator Suite (BDGS), an open source tool is used to generate synthetic data sets based on raw data sets [23].…”

Section: Workloadsmentioning

confidence: 99%

Identifying the potential of near data processing for apache spark

Awan

Ohara

Ayguadé

et al. 2017

Proceedings of the International Symposium on Memory Systems

View full text Add to dashboard Cite

While cluster computing frameworks are continuously evolving to provide real-time data analysis capabilities, Apache Spark has managed to be at the forefront of big data analytics for being a unified framework for both, batch and stream data processing. There is also a renewed interest in Near Data Processing (NDP) due to technological advancement in the last decade. However, it is not known if NDP architectures can improve the performance of big data processing frameworks such as Apache Spark. In this paper, we build the case of NDP architecture comprising programmable logic based hybrid 2D integrated processing-in-memory and instorage processing for Apache Spark, by extensive profiling of Apache Spark based workloads on Ivy Bridge Server.

show abstract

Section: Workloadsmentioning

confidence: 99%

Identifying the potential of near data processing for apache spark

Awan

Ohara

Ayguadé

et al. 2017

Proceedings of the International Symposium on Memory Systems

View full text Add to dashboard Cite

show abstract

“…Wu et al [14] name several data mining approaches with need for research to address these issues, including: Feature selection and unsupervised learning methods for sparsity, error-aware data mining for uncertainty, and data imputation methods for incompleteness. Another research direction in connection with the quality of input data is the generation of authentic synthetic data, which are needed to benchmark different Big Data solutions, for instance [15]. 10) Do Big Data lead to better results?…”

Section: ) How Can Manipulations Of Input Data Be Detected?mentioning

confidence: 99%

Trust and Big Data: A Roadmap for Research

Sänger

Richthammer

Hassan

et al. 2014

2014 25th International Workshop on Database and Expert Systems Applications

View full text Add to dashboard Cite

Abstract-We are currently living in the age of Big Data coming along with the challenge to grasp the golden opportunities at hand. This mixed blessing also dominates the relation between Big Data and trust. On the one side, large amounts of trustrelated data can be utilized to establish innovative data-driven approaches for reputation-based trust management. On the other side, this is intrinsically tied to the trust we can put in the origins and quality of the underlying data. In this paper, we address both sides of trust and Big Data by structuring the problem domain and presenting current research directions and interdependencies. Based on this, we define focal issues which serve as future research directions for the track to our vision of Next Generation Online Trust within the FORSEC project.

show abstract

“…Beyond the benchmarks for transactional processing systems, dozens of benchmarks have been proposed for big analytical processing systems, such as CloudSuite [21], BigBench [22], DCBench [23], Hibench [24], GraySort [25], CloudRank-D [26], and several other research work in [27]- [30], and so on. These benchmarks supply a set of analytical jobs or OLAP queries to test the performance to MapReduce-style systems.…”

Section: A Big Data Systems Benchmarksmentioning

confidence: 99%

Towards realistic benchmarking for cloud file systems: Early experiences

Ren

Shi

Wan

2014

2014 IEEE International Symposium on Workload Characterization (IISWC)

View full text Add to dashboard Cite

Abstract-Over the past few years, cloud file systems such as Google File System (GFS) and Hadoop Distributed File System (HDFS) have received a lot of research efforts to optimize their designs and implementations. A common issue for these efforts is performance benchmarking. Unfortunately, many system researchers and engineers face challenges on making a benchmark that reflects real-life workload cases, due to the complexity of cloud file systems and vagueness of I/O workload characteristics. They could easily make incorrect assumptions about their systems and workloads, leading to the benchmark results differing from the fact.As the preliminary step for designing a realistic benchmark, we make an effort to explore the characteristics of data and I/O workload in a production environment. We collected a two-week I/O workload trace from a 2,500-node production cluster, which is one of the largest cloud platforms in Asia. This cloud platform provides two public cloud services: data storage service (DSS) and data processing service (DPS). We analyze the commonalities and individualities between both cloud services in multiple perspectives, including the request arrival pattern, request size, data population and so on. Eight key observations are highlighted from the comprehensive study, including the arrival rate of requests follows a Lognormal distribution rather than a Poisson distribution, request arrival presents multiple periodicities, cloud file systems fit partly-open model rather than purely open model or closed model. Based on the comparative analysis results, we derive several interesting implications on guiding system researchers and engineers to build a realistic benchmark on their own systems. Finally, we discuss several open issues and challenges raised on benchmarking cloud file systems.

show abstract

BDGS: A Scalable Big Data Generator Suite in Big Data Benchmarking

Cited by 68 publications

References 20 publications

Identifying the potential of near data processing for apache spark

Identifying the potential of near data processing for apache spark

Trust and Big Data: A Roadmap for Research

Towards realistic benchmarking for cloud file systems: Early experiences

Contact Info

Product

Resources

About