SLA-Based Scheduling of Spark Jobs in Hybrid Cloud Computing Environments

Islam, Muhammed Tawfiqul; Wu, Huaming; Karunasekera, Shanika; Buyya, Rajkumar

doi:10.1109/tc.2021.3075625

Cited by 19 publications

(7 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The main focus is to deploy a Spark cluster through cost‐effective executors placement among workers nodes for any cluster job, maximizing resource utilization while prioritizing jobs based on their given deadlines. This work deployed a prototype of their SLA‐Scheduler , 32 and the benefits of their approach were evaluated using applications from the BigDataBench 34 benchmark against two scheduling techniques: Spark's FIFO and Morpheus 35 . Extending their work, Islam et al 32 tackled the problem of executing Spark applications on a hybrid cloud computing cluster.…”

Section: Related Workmentioning

confidence: 99%

“…This work deployed a prototype of their SLA‐Scheduler , 32 and the benefits of their approach were evaluated using applications from the BigDataBench 34 benchmark against two scheduling techniques: Spark's FIFO and Morpheus 35 . Extending their work, Islam et al 32 tackled the problem of executing Spark applications on a hybrid cloud computing cluster. In this version, the scheduler only analyses the jobs that meet their deadlines to make a cost‐effective executor placement decision.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Optimizing computational costs of Spark for SARS‐CoV‐2 sequences comparisons on a commercial cloud

Nunes

Melo

Tadonki

et al. 2023

Concurrency and Computation

View full text Add to dashboard Cite

Cloud computing is currently one of the prime choices in the computing infrastructure landscape. In addition to advantages such as the pay-per-use bill model and resource elasticity, there are technical benefits regarding heterogeneity and large-scale configuration. Alongside the classical need for performance, for example, time, space, and energy, there is an interest in the financial cost that might come from budget constraints. Based on scalability considerations and the pricing model of traditional public clouds, a reasonable optimization strategy output could be the most suitable configuration of virtual machines to run a specific workload. From the perspective of runtime and monetary cost optimizations, we provide the adaptation of a Hadoop applications execution cost model extracted from the literature aiming at Spark applications modeled with the MapReduce paradigm. We evaluate our optimizer model executing an improved version of the Diff Sequences Spark application to perform SARS-CoV-2 coronavirus pairwise sequence comparisons using the AWS EC2's virtual machine instances. The experimental results with our model outperformed 80% of the random resource selection scenarios. By only employing spot worker nodes exposed to revocation scenarios rather than on-demand workers, we obtained an average monetary cost reduction of 35.66% with a slight runtime increase of 3.36%.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Optimizing computational costs of Spark for SARS‐CoV‐2 sequences comparisons on a commercial cloud

Nunes

Melo

Tadonki

et al. 2023

Concurrency and Computation

View full text Add to dashboard Cite

show abstract

“…Figure 2 shows a two-tier model of virtualization technology, where infrastructure providers (InPs) are responsible for allocating physical network resources to service providers (SPs), including CPU, memory, and link resources of physical network nodes. Service providers (SPs) are responsible for providing services externally using the resources allocated to them [9][10][11][12]. This model makes the virtual network with node and link constraints act on the infrastructure.…”

Section: Network Virtualization Technologymentioning

confidence: 99%

“…In equation (11), θ represents the TCAM capacity consumption factor of the intermediate node, θ ∈ ð0, 0:5Þ.…”

Section: Wireless Communications and Mobile Computingmentioning

confidence: 99%

Research on Intelligent Mapping Algorithm of Secure Virtual Network under Cloud Computing

Niu

Liang

Zhang

2022

Wireless Communications and Mobile Computing

View full text Add to dashboard Cite

In order to improve the mapping efficiency of network intelligent mapping algorithm, this paper proposes a secure virtual network intelligent mapping algorithm under cloud computing. Firstly, the network virtualization technology is analyzed, and a network virtualization example diagram is proposed. Then the energy consumption model and network load model of virtual network mapping problem are constructed to analyze the mapping reliability. Finally, the virtual network mapping strategy is optimized by cloud computing technology to realize the research on intelligent mapping algorithm of secure virtual network under cloud computing. Experimental results show that the percentage of nodes used by the proposed algorithm is much lower than that of other mapping algorithms compared, and the energy consumption is about 8100RMB/h at 60,000 unit time point, which is much lower than that of other methods, indicating that the proposed algorithm has high mapping efficiency.

show abstract

“…Existing works [8]- [12] focus primarily on SLA management in the Cloud, and only a few recent efforts [13]- [19] perform a quantitative resource analysis based on historical data [3]- [7] to predict violations on the computing continuum. To address this need, we propose a proactive SLAaware (PROS) method relying on distributed monitoring to reduce SLA violations and increase service success on the computing continuum.…”

Section: Introductionmentioning

confidence: 99%

Proactive SLA-aware Application Placement in the Computing Continuum

Samani¹,

Mehran²,

Kimovski³

et al. 2023

2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS)

View full text Add to dashboard Cite

The accelerating growth of modern distributed applications with low delivery deadlines leads to a paradigm shift towards the multi-tier computing continuum. However, the geographical dispersion, heterogeneity, and availability of the continuum resources may result in failures and quality of service degradation, significantly negating its advantages and lowering users' satisfaction. We propose in this paper a proactive application placement (PROS) method relying on distributed coordination to prevent the quality of service violations through service-level agreements on the computing continuum. PROS employs a sigmoid function with adaptive weights for the different parameters to predict the service level agreement assurance of devices based on their past credentials and current capabilities. We evaluate PROS using two application workloads with different traffic stress levels up to 90 million services on a real testbed with 600 heterogeneous instances deployed over eight geographical locations. The results show that PROS increases the success rate by 7 %-33 %, reduces the response time by 16 %-38 %, and increases the deadline satisfaction rate by 19 %-42 % compared to two related work methods. A comprehensive simulation study with 1000 devices and a workload of up to 670 million services confirm the scalability of the results.

show abstract

SLA-Based Scheduling of Spark Jobs in Hybrid Cloud Computing Environments

Cited by 19 publications

References 28 publications

Optimizing computational costs of Spark for SARS‐CoV‐2 sequences comparisons on a commercial cloud

Optimizing computational costs of Spark for SARS‐CoV‐2 sequences comparisons on a commercial cloud

Research on Intelligent Mapping Algorithm of Secure Virtual Network under Cloud Computing

Proactive SLA-aware Application Placement in the Computing Continuum

Contact Info

Product

Resources

About