Hybrid Heuristic for Scheduling Data Analytics Workflow Applications in Hybrid Cloud Environment

Rahman, Muhammad Tauhidur; Li, Xiaorong; Palit, Henry Novianus

doi:10.1109/ipdps.2011.243

Cited by 55 publications

(35 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Rahman et al [78] propose a hybrid heuristic for scheduling data analytics workflows on heterogeneous Cloud environments; a heuristic that optimises cost of workflow execution and satisfies users requirements, such as budget, deadline, and data placement.…”

Section: Data Processing and Resource Managementmentioning

confidence: 99%

See 1 more Smart Citation

Big Data computing and clouds: Trends and future directions

Assunção

Calheiros

Bianchi

et al. 2015

Journal of Parallel and Distributed Computing

674

317

View full text Add to dashboard Cite

This paper discusses approaches and environments for carrying out analytics on Clouds for Big Data applications. It revolves around four important areas of analytics and Big Data, namely (i) data management and supporting architectures; (ii) model development and scoring; (iii) visualisation and user interaction; and (iv) business models. Through a detailed survey, we identify possible gaps in technology and provide recommendations for the research community on future directions on Cloud-supported Big Data computing and analytics solutions.

show abstract

Section: Data Processing and Resource Managementmentioning

confidence: 99%

“…With a shared platform they can upload their solutions to execute on a shared infrastructure, therefore reducing operation and maintenance costs. As discussed beforehand, techniques have been proposed for resource allocation and scheduling of Big Data analytics tasks on the Cloud [72,78].…”

Section: Business Models and Non-technical Challengesmentioning

confidence: 99%

Big Data computing and clouds: Trends and future directions

Assunção

Calheiros

Bianchi

et al. 2015

Journal of Parallel and Distributed Computing

674

317

View full text Add to dashboard Cite

show abstract

“…Xu et al [11] and Mao and Humphrey [12] proposed algorithms for scheduling multiple workflows in Clouds. Rahman et al [13] proposed an algorithm for hybrid Clouds, where at least part of the resources can be used without cost and with higher level of control over their performance. Different from this approach, we focus on the scheduling of single workflows in pure Cloud environments.…”

Section: Related Workmentioning

confidence: 99%

Meeting Deadlines of Scientific Workflows in Public Clouds with Tasks Replication

Calheiros

Buyya

2014

IEEE Trans. Parallel Distrib. Syst.

206

118

View full text Add to dashboard Cite

Abstract-The elasticity of Cloud infrastructures makes them a suitable platform for execution of deadline-constrained workflow applications, because resources available to the application can be dynamically increased to enable application speedup. Existing research in execution of scientific workflows in Clouds either try to minimize the workflow execution time ignoring deadlines and budgets or focus on the minimization of cost while trying to meet the application deadline. However, they implement limited contingency strategies to correct delays caused by underestimation of tasks execution time or fluctuations in the delivered performance of leased public Cloud resources. To mitigate effects of performance variation of resources on soft deadlines of workflow applications, we propose an algorithm that uses idle time of provisioned resources and budget surplus to replicate tasks. Simulation experiments with four well-known scientific workflows show that the proposed algorithm increases the likelihood of deadlines being met and reduces the total execution time of applications as the budget available for replication increases.

show abstract

“…Other related works such as by M. Rahman et al [13] proposed a workflow management system for deploying workflow application on virtualized environments which is able to utilize resources from public clouds. However, it does not try to optimize the resource.…”

Section: Related Workmentioning

confidence: 99%

An Iterative Optimization Framework for Adaptive Workflow Management in Computational Clouds

Wang

Duan

et al. 2013

2013 12th IEEE International Conference on Trust, Security and Privacy in Computing and Communications

Self Cite

View full text Add to dashboard Cite

Abstract-As more and more data can be generated at a fasterthan-ever rate nowadays, it becomes a challenge to processing large volumes of data for complex data analysis. In order to address performance and cost issues of big data processing on clouds, we present a novel design of adaptive workflow management system which includes an SVM (Support Vector Machine) based prediction model, workflow scheduler, and iteration controls to optimize the data processing via iterative workflow tasks. We proposed a new heuristic algorithm, called Upgrade Fit, which dynamically and continuously reallocates multiple types of cloud resources to fulfill the performance and cost requirements. The iterative workflow tasks can be bursty bags of tasks to be executed repetitively for data processing. A real application of weather forecast workflow has been used to evaluate the capability of our system for large volume image data processing. Experimental system has been set up and the results indicate that the system can effectively handle multiple types of cloud resources and optimize the performance iteratively.

show abstract

Hybrid Heuristic for Scheduling Data Analytics Workflow Applications in Hybrid Cloud Environment

Cited by 55 publications

References 6 publications

Big Data computing and clouds: Trends and future directions

Big Data computing and clouds: Trends and future directions

Meeting Deadlines of Scientific Workflows in Public Clouds with Tasks Replication

An Iterative Optimization Framework for Adaptive Workflow Management in Computational Clouds

Contact Info

Product

Resources

About