A novel real-time scheduling algorithm and performance analysis of a MapReduce-based cloud

Teng, Fei; Magoulès, Frédéric; Yu, Lei; Li, Tianrui

doi:10.1007/s11227-014-1115-z

Cited by 15 publications

(7 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It also supports prioritizing the tasks which are switched off by default and need to be turned on manually. Different scheduling techniques in Hadoop include deadline-based (Teng et al, 2014), (Morton et al, 2010); locality-aware (Palanisamy et al, 2011), (Arslan et al, 2014), (Wei et al, 2015); situation-aware (Yoo & Sim 2011), delay (Zaharia et al, 2009) (Zaharia & Borthakur et al, 2010), resource-aware (Palanisamy et al, 2011); heterogeneity-aware (Zaharia et al, 2008); network-aware (Ahmad et al, 2014), (Seo et al, 2009); and environment-aware (Zaharia et al, 2008), (Chen et al, 2010) schedulers. (Douglas et al, n.d.) raises the default Hadoop to version 2.0 architecture to a new level by splitting the resource allocation and job scheduling duties from a Job-Tracker to a Resource Manager and Application Master, respectively.…”

Section: H a N I F A N D C L E Ementioning

confidence: 99%

“…The schedulers usually try to predict the near-optimal approximation of the deadline for long-running jobs. Authors (Teng et al, 2014) proposed the paused rate monotonic (PRM) scheduling algorithm that defines a certain response time for deadline constraints services of SLA and provisions coexisting service sharing in a cloud environment. It proposes a real-time scheduling technique to deliver broad formulation and theoretical analysis of Hadoop scheduling.…”

Section: Deadline Schedulersmentioning

confidence: 99%

See 1 more Smart Citation

Jargon of Hadoop MapReduce scheduling techniques: a scientific categorization

Hanif

Lee

2019

The Knowledge Engineering Review

View full text Add to dashboard Cite

Recently, valuable knowledge that can be retrieved from a huge volume of datasets (called Big Data) set in motion the development of frameworks to process data based on parallel and distributed computing, including Apache Hadoop, Facebook Corona, and Microsoft Dryad. Apache Hadoop is an open source implementation of Google MapReduce that attracted strong attention from the research community both in academia and industry. Hadoop MapReduce scheduling algorithms play a critical role in the management of large commodity clusters, controlling QoS requirements by supervising users, jobs, and tasks execution. Hadoop MapReduce comprises three schedulers: FIFO, Fair, and Capacity. However, the research community has developed new optimizations to consider advances and dynamic changes in hardware and operating environments. Numerous efforts have been made in the literature to address issues of network congestion, straggling, data locality, heterogeneity, resource under-utilization, and skew mitigation in Hadoop scheduling. Recently, the volume of research published in journals and conferences about Hadoop scheduling has consistently increased, which makes it difficult for researchers to grasp the overall view of research and areas that require further investigation. A scientific literature review has been conducted in this study to assess preceding research contributions to the Apache Hadoop scheduling mechanism. We classify and quantify the main issues addressed in the literature based on their jargon and areas addressed. Moreover, we explain and discuss the various challenges and open issue aspects in Hadoop scheduling optimizations.

show abstract

Section: H a N I F A N D C L E Ementioning

confidence: 99%

Section: Deadline Schedulersmentioning

confidence: 99%

Jargon of Hadoop MapReduce scheduling techniques: a scientific categorization

Hanif

Lee

2019

The Knowledge Engineering Review

View full text Add to dashboard Cite

show abstract

“…Brendam Jennings [11] have discussed about the various aspects of resource management challenges in cloud environment. Fei Teng [12] have proposed the Paused Rate Monotonic (PRM) algorithm. In their algorithm they have analyzed performance of hard real time tasks on Map reduce cloud environment in which number of tasks executed by dividing the computations into smaller tasks and combining the results of various tasks.…”

Section: R a Kulkarni Shpatil Nbalajimentioning

confidence: 99%

Efficient Fuzzy Based Real Time Scheduling of Workflows on Cloud

Kulkarni¹,

Patil²,

Balaji³

2019

IJEAT

View full text Add to dashboard Cite

The technological advancements in service oriented architectures have given rise to Cloud computing as a new platform in the area of distributed computing. Virtualization is a prominent technology adopted by cloud while providing pay as per usage model. Cloud provides the computing resources in terms of different type of virtual machines. VM migration is a fundamental phenomena in cloud systems while providing uninterrupted service and load balancing to its users according to service level agreement. The virtualization and migration poses many challenges and complexities because of many uncertainties. Uncertainties are the inaccurate data in the form of task parameters, variation in the VM performance etc. As the Fuzzy logic can be employed effectively in decision making with inaccurate data we propose fuzzy logic based real time scheduling algorithm to deal with the uncertainties. The proposed algorithm in this paper performs in a better way in meeting of deadlines of real time workflows under uncertain data and VM migrations.

show abstract

“…This problem is general and impacts in a number of aspects ranging from high level development models to the low-level programming infrastructures. To address this problem, some authors have explored the computational model of Hadoop and proposed scheduling models for map-reduce applications that run in clusters [20,37,38,14]. Most of them advocate for the inclusion of rate-based and deadline-based scheduling into general computing clusters.…”

Section: Real-time Support For Stream Processingmentioning

confidence: 99%

Improving the predictability of distributed stream processors

Val

Fernandez-Garcia

Wellings

et al. 2015

Future Generation Computer Systems

View full text Add to dashboard Cite

h i g h l i g h t s• Model combining stream processing technology and real-time.• Extensions to the Storm processor.• Performance evaluation of the extension on a cluster. a b s t r a c t Next generation real-time applications demand big-data infrastructures to process huge and continuous data volumes under complex computational constraints. This type of application raises new issues on current big-data processing infrastructures. The first issue to be considered is that most of current infrastructures for big-data processing were defined for general purpose applications. Thus, they set aside real-time performance, which is in some cases an implicit requirement. A second important limitation is the lack of clear computational models that could be supported by current big-data frameworks. In an effort to reduce this gap, this article contributes along several lines. First, it provides a set of improvements to a computational model called distributed stream processing in order to formalize it as a real-time infrastructure. Second, it proposes some extensions to Storm, one of the most popular stream processors. These extensions are designed to gain an extra control over the resources used by the application in order to improve its predictability. Lastly, the article presents some empirical evidences on the performance that can be expected from this type of infrastructure.

show abstract

A novel real-time scheduling algorithm and performance analysis of a MapReduce-based cloud

Abstract: International audienc

Cited by 15 publications

References 23 publications

Jargon of Hadoop MapReduce scheduling techniques: a scientific categorization

Jargon of Hadoop MapReduce scheduling techniques: a scientific categorization

Efficient Fuzzy Based Real Time Scheduling of Workflows on Cloud

Improving the predictability of distributed stream processors

Contact Info

Product

Resources

About