A Reactive Scheduling Strategy Applied On MapReduce OLAM Operators System

Ren, Shengbing; Muheto, Dieudonne

doi:10.4304/jsw.7.11.2649-2656

Cited by 3 publications

(3 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similarly, the schedulers that were suitable for real-time circumstance were proposed [14] [15]. Shengbing Ren and Dieudonne Muheto proposed strategies to schedule tasks in OLAM (Online Analytic Mining) using MapReduce [16]. Specially, Xiaoli Wang et al took full consideration of the relationship between the performance and energy consumption of servers, and proposed a new energy-efficient task scheduling model based on MapReduce, and gave the corresponding algorithm [17].…”

Section: Other Schedulersmentioning

confidence: 99%

An Optimized Algorithm for Reduce Task Scheduling

Zhang¹,

Hu²,

Jiang³

2014

JCP

View full text Add to dashboard Cite

In this paper, we propose a novel algorithm to solve the starving problem of the small jobs and reduce the process time of the small jobs on Hadoop platform. Current schedulers of MapReduce/Hadoop are quite successful in achieving data locality and scheduling the reduce tasks with a greedy algorithm. Some jobs may have hundreds of map tasks and just several reduce tasks, in which case, the reduce tasks of the large jobs require more time for waiting, which will result in the starving problem of the small jobs. Since the map tasks and the reduce tasks are scheduled separately, we can change the way the scheduler launches the reduce tasks without affecting the map phase. Therefore we develop an optimized algorithm to schedule the reduce tasks with the shortest remaining time (SRT) of the map tasks. We apply our algorithm to the fair scheduler and the capacity scheduler, which are both widely used in real production environment. The evaluation results show that the SRT algorithm can decrease the process time of the small jobs effectively.

show abstract

Section: Other Schedulersmentioning

confidence: 99%

An Optimized Algorithm for Reduce Task Scheduling

Zhang¹,

Hu²,

Jiang³

2014

JCP

View full text Add to dashboard Cite

show abstract

“…With the rise of big data, the number of Hadoop-based application and research continues to increase [1][2][3][4][5][6], and the application areas also keep expanding [7][8][9]. Conventional log data processing models cannot deal with the massive amount of log data for large network systems, so Hadoop-based log analysis systems are emerged [10][11][12][13][14].…”

Section: Introductionmentioning

confidence: 99%

Research on Cloud-Based Mass Log Data Management Mechanism

Yang¹,

Liu²,

Zhao³

2014

JCP

View full text Add to dashboard Cite

In this paper, we study the file management mechanism of large-scale cloud-based log data. With the rise of big data, there are more and more the Hadoop-based applications. Log analysis is an important part of network security management, but the existing network log analysis system can't deal with huge amounts of log data, or only use offline mode which with a longer response delay. Therefore, building the online Hadoop-based log processing system is necessary. However, how to effectively manage vast amounts of log data have become the key problems of such system. To this end, this paper puts forward a new hierarchical file archiving (HFA) mechanism which can realize the hierarchical and sorted storage of massive amounts of log data. In addition, some feasible methods for the mechanism are also proposed. Through the HFA mechanism, the traditional log analysis mode and Hadoopbased offline analysis mode can be combined to achieve the online Hadoop-based log analysis system, which have good scalability that can effectively store and handle the massive log data, and faster response speed for user request to meet the requirements of online processing. The feasibility and effectiveness of the HFA mechanism have been verified by the experiment of a small log process system.

show abstract

“…MapReduce [4], which is first proposed by Google in 2004, has become the dominant distributed parallel programming paradigm to solve the data-intensive problems in the cloud computing domain. MapReduce [21] paradigm is built on the top of the Goolge File System (GFS) [5]. It processes the data with the form of key-value pairs in a cluster of commodity machines.…”

Section: Introductionmentioning

confidence: 99%

A Scalable XSLT Processing Framework based on MapReduce

Li¹,

Luo²,

Yang³

et al. 2013

JCP

View full text Add to dashboard Cite

The eXtensible Stylesheet Language Transformation (XSLT) is a de-facto standard for XML data transforming and extracting. Efficient processing of large amounts of XML data brings challenges to conventional XSLT processors, which are designed to run in a single machine context. To solve these data-intensive problems, MapReduce paradigm in the cloud computing domain has received a comprehensive attention in both academia and IT industry recently. In this paper, a novel MapReduce-based XSLT distributed processing framework named CloudXSLT is proposed to implement efficient and scalable XML data transforming. First, the architecture of CloudXSLT framework is outlined. Subsequently, several XML data and XSLT rule representation models which are suitable for MapReduce paradigm are defined, and several MapReduce-based XSLT distributed processing algorithms are proposed. Finally, an experiment on a simulation environment with real XML datasets shows our framework is more efficient and scalable than conventional XSLT processors when processing large size of XML data.

show abstract

A Reactive Scheduling Strategy Applied On MapReduce OLAM Operators System

Cited by 3 publications

References 25 publications

An Optimized Algorithm for Reduce Task Scheduling

An Optimized Algorithm for Reduce Task Scheduling

Research on Cloud-Based Mass Log Data Management Mechanism

A Scalable XSLT Processing Framework based on MapReduce

Contact Info

Product

Resources

About