Fast and Lightweight Execution Time Predictions for Spark Applications

Amannejad, Yasaman; Shah, Sarah; Krishnamurthy, Diwakar; Wang, Mea

doi:10.1109/cloud.2019.00088

Cited by 8 publications

(10 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several studies and modeling exercises have been conducted on the scalability of batch analytics workloads, e.g. [34]- [41]. Since batch and streaming workloads function quite differently, we will discuss the relevant parts of these studies throughout the results section.…”

Section: Related Workmentioning

confidence: 99%

Influencing Factors in the Scalability of Distributed Stream Processing Jobs

Dongen

Poel

2021

IEEE Access

View full text Add to dashboard Cite

More and more use cases require fast, accurate, and reliable processing of large volumes of data. To do this, a distributed stream processing framework is needed which can distribute the load over several machines. In this work, we study and benchmark the scalability of stream processing jobs in four popular frameworks: Flink, Kafka Streams, Spark Streaming, and Structured Streaming. Besides that, we determine the factors that influence the performance and efficiency of scaling processing jobs with distinct characteristics. We evaluate horizontal, as well as vertical scalability. Our results show how the scaling efficiency is impacted by many factors including the initial cluster layout and direction of scaling, the pipeline design, the framework design, resource allocation, and data characteristics. Finally, we give some recommendations on how practitioners should undertake to scale their clusters.

show abstract

Section: Related Workmentioning

confidence: 99%

Influencing Factors in the Scalability of Distributed Stream Processing Jobs

Dongen

Poel

2021

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Amannejad et al [28] proposed an approach for Spark execution time prediction with less prior executions of the applications based on Amdahl's law [15]. This approach is capable of predicting the execution time within a short period.…”

Section: Related Workmentioning

confidence: 99%

“…One of the limitations of this work is that they validated this approach only with a single node cluster, not on a real cluster environment. Amannejad and Shah extended their previous work [28] and proposed an alternative model called PERIDOT [29] for quick execution time prediction with limited cluster resource settings and a small subset of input data. They analysed the logs from both of the executions and checked the internal dependencies between the internal stages.…”

Section: Related Workmentioning

confidence: 99%

A parallelization model for performance characterization of Spark Big Data jobs on Hadoop clusters

et al. 2021

View full text Add to dashboard Cite

This article proposes a new parallel performance model for different workloads of Spark Big Data applications running on Hadoop clusters. The proposed model can predict the runtime for generic workloads as a function of the number of executors, without necessarily knowing how the algorithms were implemented. For a certain problem size, it is shown that a model based on serial boundaries for a 2D arrangement of executors can fit the empirical data for various workloads. The empirical data was obtained from a real Hadoop cluster, using Spark and HiBench. The workloads used in this work were included WordCount, SVM, Kmeans, PageRank and Graph (Nweight). A particular runtime pattern emerged when adding more executors to run a job. For some workloads, the runtime was longer with more executors added. This phenomenon is predicted with the new model of parallelisation. The resulting equation from the model explains certain performance patterns that do not fit Amdahl’s law predictions, nor Gustafson’s equation. The results show that the proposed model achieved the best fit with all workloads and most of the data sizes, using the R-squared metric for the accuracy of the fitting of empirical data. The proposed model has advantages over machine learning models due to its simplicity, requiring a smaller number of experiments to fit the data. This is very useful to practitioners in the area of Big Data because they can predict runtime of specific applications by analysing the logs. In this work, the model is limited to changes in the number of executors for a fixed problem size.

show abstract

“…Amannejad et al [13] proposed a model to quick prediction runtime a job when little cluster resources are available. They used logs from two executions of an application with small sample data and different resource settings and explore the accuracy of the predictions for other resource allocation settings and input data sizes then expanded their model to the Spark waves.…”

Section: Background and Related Workmentioning

confidence: 99%

“…(27), Figs. 11,12,13,14 shows the error rate of the estimated runtime by the Exponential Averaging method and actual runtime in WordCount, Tera-Sort, and Inverted index in the different number of nodes. The results show that the error rate is less than 5%.…”

Section: Evaluation Of Estimating Runtime Of a Job That Has Already Bmentioning

confidence: 99%

Estimating runtime of a job in Hadoop MapReduce

Peyravi

Moeini

2020

J Big Data

View full text Add to dashboard Cite

Nowadays, with the emergence and use of new systems, we face a massive amount of data. Due to the volume, velocity, and variety of these big data, managing, maintaining, and processing them require special infrastructures. One of the famous open-source frameworks is Apache Hadoop [1]. It is a scalable and reliable framework for storage and process big data. Hadoop divides the big input data into fixed-size pieces; stores and processes these split of data on a cluster of machines. By default, each split copies three times and transfer to different machines to manage errors and fault tolerance. Hadoop stores its data in a distributed file system called HDFS. MapReduce is designed to work with HDFS. MapReduce is the programming model that allows Hadoop to efficiently process large amounts of data in the cluster's nodes. [1-5]. Since one of Hadoop's most important tasks is managing jobs and resources, better management will be done if estimation and prediction the runtime of a job do precisely. Also, because of limited critical resources like CPU, I/O, and memory, this issue is important in many aspects like efficient scheduling, better energy consumption, bottleneck detection, and resource management [3-5].

show abstract

Fast and Lightweight Execution Time Predictions for Spark Applications

Cited by 8 publications

References 4 publications

Influencing Factors in the Scalability of Distributed Stream Processing Jobs

Influencing Factors in the Scalability of Distributed Stream Processing Jobs

A parallelization model for performance characterization of Spark Big Data jobs on Hadoop clusters

Estimating runtime of a job in Hadoop MapReduce

Contact Info

Product

Resources

About