Quick Execution Time Predictions for Spark Applications

Shah, Sarah; Amannejad, Yasaman; Krishnamurthy, Diwakar; Wang, Mea

doi:10.23919/cnsm46954.2019.9012752

Cited by 22 publications

(17 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There have been several efforts on modeling the performance of Spark applications [12], [16]- [22] in the absence of interference. These approaches use historical executions of an application to derive a model that can predict the application's execution time under various configurations, e.g., different input data sizes and core allocations.…”

Section: Related Workmentioning

confidence: 99%

“…A stage consists of a set of tasks with each task applying the operations of the stage on one partition of the data. Spark stages may be variable, i.e., displaying an increase in the number of tasks with an increase in input data, or they could be constant, with the same number of tasks irrespective of input data size [12]. A task is executed by one of the executors allocated by Spark to that stage.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Diaspore: Diagnosing Performance Interference in Apache Spark

2021

Self Cite

View full text Add to dashboard Cite

Apache Spark is being increasingly used to execute big data applications on cluster computing platforms. To increase system utilization, cluster operators often configure their clusters such that multiple co-located applications can simultaneously share the resources of a cluster node. With resource sharing, applications can compete with each other for shared node resources thereby interfering with each other's performance. Many Spark applications take a long time to execute. Performance interference from other applications can thus cause a Spark application to fail or take even longer time to execute thereby wasting cluster resources and frustrating users. This motivates the need for an automated technique that can detect interference quickly and also diagnose the root cause of the interference to facilitate mitigation of the problem. Most existing approaches are not designed to offer quick interference detection and diagnosis. For example, they typically require extensive training data for every application of interest under various possible input data sizes and resource allocations. In this paper, we systematically investigate the design of a Machine Learning (ML) based technique that addresses this open problem. We implement a tool called Diaspore that integrates our findings. We evaluate the tool with a diverse set of 13 Spark applications executing on a real cluster. Experimental results show that Diaspore requires only small scale training data, i.e., executions under small input sizes and resource allocations. Furthermore, our results show that the tool can offer accurate predictions for applications not present in the training data. Consequently, Diaspore reduces the training time needed to offer predictions. Finally, the feature engineering underlying Diaspore ensures that the tool can detect and diagnose interference quickly in an online manner by sampling only a small fraction of a long running application's execution. This can allow cluster operators to mitigate interference in an agile manner.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Diaspore: Diagnosing Performance Interference in Apache Spark

2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…These workload and system parameters are discussed in Section 4.1.2. These workload and system parameters are observed to have a significant effect on the computation time, speedup, and efficiency by other researchers [58]. The selection of the appropriate number of worker nodes and the number of executor cores per worker node heavily influences the performance of the Filter method.…”

Section: Performance Evaluation Of the Filter Methods For Use Case-1mentioning

confidence: 99%

“…The numbers of executor cores used in the experiments are 1, 2, 4, 6, 8, and 12 where the default value is set to 8. The research presented in[58] used up to 40 executor cores where each worker node has 5 executor cores. The master node of Spark uses 1 core.…”

mentioning

confidence: 99%

See 1 more Smart Citation

A Parallel Processing Technique for Filtering and Storing User Specified Data

Chanda¹

View full text Add to dashboard Cite

Users are often interested in a specific type of data (user-preferred data) from a largevolume dataset. An efficient system that only stores user-preferred data from the large dataset can reduce the search latency, which allows the users to search for relevant information in a timely manner. The motivation behind this thesis is to devise a technique that filters a large dataset and stores only the filtered data, thereby saving storage space for the user. Running the filtering operation can be CPU-intensive, which can lead to high latency in extracting preferred data from the dataset. To solve this problem, the technique employs parallel processing and machine learning. A proof-of-concept prototype for this technique has been built on Apache Spark. The performance of the prototype subjected to synthetic datasets is analyzed. The analysis of experimental results shows the viability of this technique and provides insights into the system behavior and performance.

show abstract

A Deterministic Model to Predict Execution Time of Spark Applications

Tariq

Das

2023

Computer Performance Engineering

View full text Add to dashboard Cite

Quick Execution Time Predictions for Spark Applications

Cited by 22 publications

References 9 publications

Diaspore: Diagnosing Performance Interference in Apache Spark

Diaspore: Diagnosing Performance Interference in Apache Spark

A Parallel Processing Technique for Filtering and Storing User Specified Data

A Deterministic Model to Predict Execution Time of Spark Applications

Contact Info

Product

Resources

About