Deadline-Aware Cost Optimization for Spark

Sidhanta, Subhajit; Golab, Wojciech; Mukhopadhyay, Supratik

doi:10.1109/tbdata.2019.2908188

Cited by 12 publications

(15 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As it can be observed via Table 4, several works predict execution time [13]- [15], [17], [31], [33], [63], while others estimate memory consumption [16], [18], [29]. It can be observed that in [13]- [18], [63], the prediction is done for big data workloads, whereas in [29], [31], [33], the considered applications are routine ones. In [14], a performance prediction framework Ernest is proposed, which can predict the execution time on a hardware configuration, given a job and its input.…”

Section: Related Workmentioning

confidence: 99%

“…In [13], a machine learning approach is proposed for predicting the execution time of Spark applications. Similarly, an empirical model is proposed in [63], for estimating the completion time of the Spark job on a cloud, with respect to cluster nodes count, input data size, and the number of iterations. The model predicts the cost optimal cluster configuration for executing the Spark job on a cloud under the SLO (Service Level Objective) specified completion deadline.…”

Section: Related Workmentioning

confidence: 99%

“…In comparison to existing works, the proposed work appears to be the first framework that successfully predicts all three big data metrics. Additionally, the prediction models proposed in [13]- [18], [63] work for a specific platform like Spark and MapReduce. Whereas, similar to [29], [31], [33], the proposed prediction model is generic because it is implemented at the compiler stage.…”

Section: Related Workmentioning

confidence: 99%

“…Additionally, the proposed model has been tested with different types of datasets like text, graph, and image. Whereas, mostly existing solutions [13], [14], [18], [29], [31], [33] are designed for text datasets only, [17] for image dataset, few [15], [16] for text and graph both, while only [63] considers graph, image, and text. It can be observed via Table 4, the metrics predictions in existing works have been done in a different environment, using different techniques and workloads, hence a direct comparison of these with proposed work is not feasible.…”

Section: Related Workmentioning

confidence: 99%

See 3 more Smart Citations

Towards a Novel Framework for Automatic Big Data Detection

Ahmed

Ismail

2020

IEEE Access

View full text Add to dashboard Cite

Big data is a "relative" concept. It is the combination of data, application, and platform properties. Recently, big data specific technologies have emerged, including software frameworks, databases, hardware accelerators, storage technologies, etc. However, the automatic selection of these solutions for big data computations remains a non-trivial task. Presently, the big data tools are selected by analyzing the problem manually, or by using several performance prediction techniques. The manual identification is based on the data properties only, whereas the performance predictors only estimate basic execution metrics without linking them with big data (3Vs) thresholds. Hence, both ways of identification are mostly incorrect, which can lead to inefficient use of 3Vs optimizations, resulting into global inefficiency, reduced system performance, increasing power consumption, requiring greater effort on the part of the programming team, and misallocation of the hardware resources required for the task. In this regard, a novel framework has been proposed for automatic detection of 3Vs (Volume, Velocity, Variety) of big data, using machine learning. The detection is done through static code features, data, and platform properties, leading to relevant tool selection, and code generation, with minimal overheads, lesser programmer interventions, higher usability, and portability. Instead of handling each application with big data specialized solutions, or manually identifying the 3Vs, the framework can automatically detect and link the 3Vs to the relevant optimizations. Several standard applications have been tested using the proposed framework. In the case of volume, the average detection accuracy is up to 97.8% for seen and 95.9% for unseen applications. In the case of velocity, the average detection accuracy is up to 97.3% for seen and 92.6% for unseen applications. There is no margin of error in variety detection, as it has straightforward computations without any predictions. Furthermore, an airline recommendation system case study strengthens the effectiveness of the proposed approach.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Towards a Novel Framework for Automatic Big Data Detection

Ahmed

Ismail

2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…To optimize T DP , modifications were applied to the Apache Spark Streaming process to satisfy the real-time delay bounds even when errors occur in the communication and data computing process. The SRSAF scheme proposed in this paper is different from the OptEx [25] and Spark adaptive failure-compensation (SAF) [26] schemes because of how it was designed to cope with errors in the communication and data computing process while satisfying the real-time delay bounds that support control of the UAVs in the HAP and LAP as well as the network data services.…”

Section: Optimization Of Uav Control Communication and Data Processingmentioning

confidence: 99%

Combined time bound optimization of control, communication, and data processing for FSO‐based 6G UAV aerial networks

Seo

Chung

2020

ETRI Journal

View full text Add to dashboard Cite

Because of the rapid increase of mobile traffic, flexible broadband supportive unmanned aerial vehicle (UAV)‐based 6G mobile networks using free space optical (FSO) links have been recently proposed. Considering the advancements made in UAVs, big data processing, and artificial intelligence precision control technologies, the formation of an additional wireless network based on UAV aerial platforms to assist the existing fixed base stations of the mobile radio access network is considered a highly viable option in the near future. In this paper, a combined time bound optimization scheme is proposed that can adaptively satisfy the control and communication time constraints as well as the processing time constraints in FSO‐based 6G UAV aerial networks. The proposed scheme controls the relation between the number of data flows, input data rate, number of worker nodes considering the time bounds, and the errors that occur during communication and data processing. The simulation results show that the proposed scheme is very effective in satisfying the time constraints for UAV control and radio access network services, even when errors in communication and data processing may occur.

show abstract

Adaptive Trust Management and Data Process Time Optimization for Real-Time Spark Big Data Systems

Seo

Chung²

2021

IEEE Access

View full text Add to dashboard Cite

Deadline-Aware Cost Optimization for Spark

Cited by 12 publications

References 26 publications

Towards a Novel Framework for Automatic Big Data Detection

Towards a Novel Framework for Automatic Big Data Detection

Combined time bound optimization of control, communication, and data processing for FSO‐based 6G UAV aerial networks

Adaptive Trust Management and Data Process Time Optimization for Real-Time Spark Big Data Systems

Contact Info

Product

Resources

About