Efficient Performance Prediction for Apache Spark

Cheng, Guoli; Song, Ying; Wang, Bingming; Li, Yuhang

doi:10.1016/j.jpdc.2020.10.010

Cited by 31 publications

(13 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…e method was evaluated using six standard Spark benchmarks, each with five distinct input datasets. e findings indicated that the average error of the model built using the proposed method was just 9.02 percent, which was much less than the average error of the current approaches [14].…”

Section: Regression Algorithms Using Sparkmentioning

confidence: 74%

Usages of Spark Framework with Different Machine Learning Algorithms

Mohamed

El-Henawy

Salah

2021

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

Sensors, satellites, mobile devices, social media, e-commerce, and the Internet, among others, saturate us with data. The Internet of Things, in particular, enables massive amounts of data to be generated more quickly. The Internet of Things is a term that describes the process of connecting computers, smart devices, and other data-generating equipment to a network and transmitting data. As a result, data is produced and updated on a regular basis to reflect changes in all areas and activities. As a consequence of this exponential growth of data, a new term and idea known as big data have been coined. Big data is required to illuminate the relationships between things, forecast future trends, and provide more information to decision-makers. The major problem at present, however, is how to effectively collect and evaluate massive amounts of diverse and complicated data. In some sectors or applications, machine learning models are the most frequently utilized methods for interpreting and analyzing data and obtaining important information. On their own, traditional machine learning methods are unable to successfully handle large data problems. This article gives an introduction to Spark architecture as a platform that machine learning methods may utilize to address issues regarding the design and execution of large data systems. This article focuses on three machine learning types, including regression, classification, and clustering, and how they can be applied on top of the Spark platform.

show abstract

Section: Regression Algorithms Using Sparkmentioning

confidence: 74%

Usages of Spark Framework with Different Machine Learning Algorithms

Mohamed

El-Henawy

Salah

2021

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

show abstract

“…Cheng [25] proposed a performance model based on Adaboost at stage-level for Spark runtime prediction. They considered a classic projective sampling and data mining technique such as projective sampling and advanced sampling to reduce the model's overhead.…”

Section: Related Workmentioning

confidence: 99%

A parallelization model for performance characterization of Spark Big Data jobs on Hadoop clusters

et al. 2021

View full text Add to dashboard Cite

This article proposes a new parallel performance model for different workloads of Spark Big Data applications running on Hadoop clusters. The proposed model can predict the runtime for generic workloads as a function of the number of executors, without necessarily knowing how the algorithms were implemented. For a certain problem size, it is shown that a model based on serial boundaries for a 2D arrangement of executors can fit the empirical data for various workloads. The empirical data was obtained from a real Hadoop cluster, using Spark and HiBench. The workloads used in this work were included WordCount, SVM, Kmeans, PageRank and Graph (Nweight). A particular runtime pattern emerged when adding more executors to run a job. For some workloads, the runtime was longer with more executors added. This phenomenon is predicted with the new model of parallelisation. The resulting equation from the model explains certain performance patterns that do not fit Amdahl’s law predictions, nor Gustafson’s equation. The results show that the proposed model achieved the best fit with all workloads and most of the data sizes, using the R-squared metric for the accuracy of the fitting of empirical data. The proposed model has advantages over machine learning models due to its simplicity, requiring a smaller number of experiments to fit the data. This is very useful to practitioners in the area of Big Data because they can predict runtime of specific applications by analysing the logs. In this work, the model is limited to changes in the number of executors for a fixed problem size.

show abstract

“…For example, Jyothi et al [22] analyzed workloads on production clusters and showed that about 40% of applications were non-recurrent, i.e., did not have historical execution data. Cheng et al [7] reduce the training samples and lower the model overhead in their Adaboost-based performance prediction model by utilizing projective sampling, however their process still requires a significant amount of training.…”

Section: Apache Spark Platformmentioning

confidence: 99%

PERIDOT: Modeling Execution Time of Spark Applications

Shah¹,

Amannejad²,

Krishnamurthy³

et al. 2021

IEEE Open J. Comput. Soc.

View full text Add to dashboard Cite

A data analytics application submitted to a Spark cluster often has to finish executing by a specified time target. To use cluster resources effectively, the key challenge is having the ability to gain quick insights on how the execution time of any given application is likely to be impacted by the resources allocated to the application, e.g., the number of Spark executor cores assigned, and the size of the data to be processed. Such insights can be used to quickly estimate the required resources and configure a Spark application for a desired execution time using the least amount of resources. Our paper proposes an automated execution time estimation approach called PERIDOT that involves executing a given application under a fixed resource setting with two different-sized, small subsets of its input data to offer fast, lightweight execution time predictions. It analyzes logs from these two executions to estimate the dependencies between internal stages of the application. Information on these dependencies combined with knowledge of Spark's data partitioning mechanisms is used to derive an analytic model that can estimate execution times for other resource settings and input data sizes. Our results from a wide range of applications and multiple Spark clusters show that PERIDOT can accurately estimate the execution time of an application from limited historical data, and suggest the minimum amount of resources required to closely meet an execution time target.

show abstract

Efficient Performance Prediction for Apache Spark

Cited by 31 publications

References 11 publications

Usages of Spark Framework with Different Machine Learning Algorithms

Usages of Spark Framework with Different Machine Learning Algorithms

A parallelization model for performance characterization of Spark Big Data jobs on Hadoop clusters

PERIDOT: Modeling Execution Time of Spark Applications

Contact Info

Product

Resources

About