dSpark: Deadline-Based Resource Allocation for Big Data Applications in Apache Spark

Islam, Muhammed Tawfiqul; Karunasekera, Shanika; Buyya, Rajkumar

doi:10.1109/escience.2017.21

Cited by 28 publications

(16 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [15] a mathematical model of Spark is proposed, where the execution time of Spark is determined by a stage and partition unit. Sidhanta et al [16] and Islam et al [17] mathematically model the Spark execution time by proposing OptEx [16] and dSpark [17], which are resource allocation policies that find the lowest cost while satisfying the deadline. OptEx [16] and dSpark [17] show an accuracy in the range of 95-97%.…”

Section: B Apache Spark System Modelmentioning

confidence: 99%

“…Sidhanta et al [16] and Islam et al [17] mathematically model the Spark execution time by proposing OptEx [16] and dSpark [17], which are resource allocation policies that find the lowest cost while satisfying the deadline. OptEx [16] and dSpark [17] show an accuracy in the range of 95-97%. However, these models do not take into account the influence of errors and failures, which can lead to a timeout that may result in deadline violations.…”

Section: B Apache Spark System Modelmentioning

confidence: 99%

“…Although many studies have been carried out on Spark and Hadoop systems, analysis of the existing prediction models reveals that there are a couple of important yet unsolved issues. First, prediction models are mostly based on environments where failures did not occur [3], [15], [16], [17], and second, even for the models that considered failures, a probability of failure occurrence (HP model) was not included into the resource provisioning process [3]. Because failures do occur in real world systems, when applying existing time prediction models in resource provisioning, target completion time violations frequently occur.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Time Estimation and Resource Minimization Scheme for Apache Spark and Hadoop Big Data Systems With Failures

2019

View full text Add to dashboard Cite

Apache Spark and Hadoop are open source frameworks for big data processing, which have been adopted by many companies. In order to implement a reliable big data system that can satisfy processing target completion times, accurate resource provisioning and job execution time estimations are needed. In this paper, time estimation and resource minimization schemes for Spark and Hadoop systems are presented. The proposed models use the probability of failure in the estimations to more accurately formulate the characteristics of real big data operations. The experimental results show that the proposed Spark adaptive failure-compensation and Hadoop adaptive failure-compensation schemes improve the accuracy of resource provisions by considering failure events, which improves the scheduling success rate of big data processing tasks.

show abstract

Section: B Apache Spark System Modelmentioning

confidence: 99%

Section: B Apache Spark System Modelmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Time Estimation and Resource Minimization Scheme for Apache Spark and Hadoop Big Data Systems With Failures

2019

View full text Add to dashboard Cite

show abstract

“…Resource Provisioning in Cloud. There have been extensive research works [9,16,18] by cloud community on resource provisioning. These works focus more on deciding the number of machines to process an application.…”

Section: Related Workmentioning

confidence: 99%

“…Furthermore, the idle tasks also waste the computational resources which are assigned to them. The latter is not considered even in the area of cloud computing [9,16,18], where computational resources are decided based on the total data size. This leads to wastage of resources and money.…”

Section: Introductionmentioning

confidence: 99%

Automatically Configuring Parallelism for Hybrid Layouts

Munir

Abelló

Romero

et al. 2019

Communications in Computer and Information Science

View full text Add to dashboard Cite

Distributed processing frameworks process data in parallel by dividing it into multiple partitions and each partition is processed in a separate task. The number of tasks is always created based on the total file size. However, this can lead to launch more tasks than needed in the case of hybrid layouts, because they help to read less data for certain operations (i.e., projection, selection). The over-provisioning of tasks may increase the job execution time and induce significant waste of computing resources. The latter due to the fact that each task introduces extra overhead (e.g., initialization, garbage collection, etc.). To allow a more efficient use of resources and reduce the job execution time, we propose a cost-based approach that decides the number of tasks based on the data being read. The proposed cost-model can be utilized in a multi-objective approach to decide both the number of tasks and number of machines for execution.

show abstract

Exponentially‐spider monkey optimization based allocation of resource in cloud

Kalpana

Prabhu²,

Polepally

et al. 2021

Int J of Intelligent Sys

View full text Add to dashboard Cite

The increasing desire for distributed computing systems has attracted huge interest in memory and computing resources. The cloud provides on-demand access to provide a flexible allocation of resources for reliable services. Therefore, there should be a provision in which resources are accessible to request users to satisfy user needs. In classical techniques, the allocation of resources by satisfying power and Quality-of-Services is a challenging aspect. This paper devises a novel technique for optimal resource allocation, namely, Exponentially Spider Monkey Optimization (E-SMO). Here, the proposed E-SMO is devised by combining Exponential Weighted Moving Average and Spider Monkey Optimization (SMO). Besides, the fitness function is newly devised considering resource utilization and resource cost. After that, the cloud resources employ a switching strategy to reduce power consumption to prevent the switching of redundant servers. For optimal switching, the proposed E-SMO is utilized with other fitness factors that compute the number of applications assigned in the physical machine using the switching state is in OFF condition. Thus, the server switching model is incorporated to activate or deactivate the server when not in use for effective resources utilization. The proposed E-SMO algorithm outperformed other methods with the maximal resource

show abstract

dSpark: Deadline-Based Resource Allocation for Big Data Applications in Apache Spark

Cited by 28 publications

References 17 publications

Time Estimation and Resource Minimization Scheme for Apache Spark and Hadoop Big Data Systems With Failures

Time Estimation and Resource Minimization Scheme for Apache Spark and Hadoop Big Data Systems With Failures

Automatically Configuring Parallelism for Hybrid Layouts

Exponentially‐spider monkey optimization based allocation of resource in cloud

Contact Info

Product

Resources

About