A self-adaptive scheduling algorithm for reduce start time

Tang, Zhuo; Jiang, Lingang; Zhou, Junqing; Li, Kenli; Li, Keqin

doi:10.1016/j.future.2014.08.011

Cited by 59 publications

(24 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…These studies primarily optimize scheduling algorithms to reduce the associated time allocation using heuristic algorithms and intelligent optimization algorithms, such as in four previous studies [9], [12], [22], [23]. The first of these studies involved a super-heuristic algorithm whose target was to achieve optimal span [9].…”

Section: B Focus On Timementioning

confidence: 99%

A Multiqueue Interlacing Peak Scheduling Method Based on Tasks’ Classification in Cloud Computing

Zuo

Dong

Shu

et al. 2018

IEEE Systems Journal

View full text Add to dashboard Cite

Abstract-In cloud computing, resources are dynamic, and the demands placed on the resources allocated to a particular task are diverse. These factors could lead to load imbalances, which affect scheduling efficiency and resource utilization. A scheduling method called interlacing peak is proposed. First, the resource load information, such as CPU, I/O, and memory usage, is periodically collected and updated, and the task information regarding CPU, I/O, and memory is collected. Second, resources are sorted into three queues according to the loads of the CPU, I/O, and memory: CPU intensive, I/O intensive, and memory intensive, according to their demands for resources. Finally, once the tasks have been scheduled, they need to interlace the resource load peak. Some types of tasks need to be matched with the resources whose loads correspond to a lighter types of tasks. In other words, CPUintensive tasks should be matched with resources with low CPU utilization; I/O-intensive tasks should be matched with resources with shorter I/O wait times; and memory-intensive tasks should be matched with resources that have low memory usage. The effectiveness of this method is proved from the theoretical point of view. It has also been proven to be less complex in regard to time and place. Four experiments were designed to verify the performance of this method. Experiments leverage four metrics: 1) average response time; 2) load balancing; 3) deadline violation rates; and 4) resource utilization. The experimental results show that this method can balance loads and improve the effects of resource allocation and utilization effectively. This is especially true when resources are limited. In this way, many tasks will compete for the same resources. However, this method shows advantage over other similar standard algorithms.

show abstract

Section: B Focus On Timementioning

confidence: 99%

A Multiqueue Interlacing Peak Scheduling Method Based on Tasks’ Classification in Cloud Computing

Zuo

Dong

Shu

et al. 2018

IEEE Systems Journal

View full text Add to dashboard Cite

show abstract

“…Then, it dynamically selects one of the heuristic algorithms to solve the scheduling problem. In [8], the authors considered information of tasks (i.e., task completion time and size of tasks) and the information is used in the self-adaptive scheduling technique to reduce total processing time and average response time. To solve dependency issues of cloud tasks, the scheduling problem can be translated to the directed acyclic graph (DAG) [9].…”

Section: Categorymentioning

confidence: 99%

A slave ants based ant colony optimization algorithm for task scheduling in cloud computing environments

Moon

Gil

et al. 2017

Hum. Cent. Comput. Inf. Sci.

View full text Add to dashboard Cite

Since cloud computing provides computing resources on a pay per use basis, a task scheduling algorithm directly affects the cost for users. In this paper, we propose a novel cloud task scheduling algorithm based on ant colony optimization that allocates tasks of cloud users to virtual machines in cloud computing environments in an efficient manner. To enhance the performance of the task scheduler in cloud computing environments with ant colony optimization, we adapt diversification and reinforcement strategies with slave ants. The proposed algorithm solves the global optimization problem with slave ants by avoiding long paths whose pheromones are wrongly accumulated by leading ants.

show abstract

“…SARS [32] is another optimal scheduling policy that manages reduce tasks' start times in Hadoop. It decreases the completion time of reduce tasks by deciding the start time of each reduce task dynamically according to each job context, such as the task completion time and the size of map output.…”

Section: Related Workmentioning

confidence: 99%

iiHadoop: an asynchronous distributed framework for incremental iterative computations

Saadon

Mokhtar

2017

J Big Data

View full text Add to dashboard Cite

IntroductionToday, a large amount of data is being produced in many areas including: e-commerce, social network, finance, health-care, and education. This increase in data volume consequently increases the need for an efficient computing framework to process this data and transform it into meaningful information. In the past years, many distributed computing frameworks [1][2][3][4][5][6] have been developed to perform large-scale data processing. MapReduce [2] (with its open-source implementation, Hadoop [7]) is the most widely used framework because of its simplicity, scalability, efficiency, and reliability. It was proposed by Google in 2004 to enable the distribution and processing of large-scale data over a cluster of commodity machines. The framework takes care of the distributed execution, fault tolerance, load balancing, and job scheduling; therefore, developers and programmers do not need to worry about these issues and can simply perform their tasks. AbstractIt is true that data is never static; it keeps growing and changing over time. New data is added and old data can either be modified or deleted. This incremental nature of data motivates the development of new systems to perform large-scale data computations incrementally. MapReduce was recently introduced to provide an efficient approach for handling large-scale data computations. Nevertheless, it turned to be inefficient in supporting the processing of small incremental data. While many previous systems have extended MapReduce to perform iterative or incremental computations, these systems are still inefficient and too expensive to perform large-scale iterative computations on changing data. In this paper, we present a new system called iiHadoop, an extension of Hadoop framework, optimized for incremental iterative computations. iiHadoop accelerates program execution by performing the incremental computations on the small fraction of data that is affected by changes rather than the whole data. In addition, iiHadoop improves the performance by executing iterations asynchronously, and employing locality-aware scheduling for the map and reduce tasks taking into account the incremental and iterative behavior. An evaluation for the proposed iiHadoop framework is presented using examples of iterative algorithms, and the results showed significant performance improvements over comparable existing frameworks.

show abstract

A self-adaptive scheduling algorithm for reduce start time

Cited by 59 publications

References 30 publications

A Multiqueue Interlacing Peak Scheduling Method Based on Tasks’ Classification in Cloud Computing

A Multiqueue Interlacing Peak Scheduling Method Based on Tasks’ Classification in Cloud Computing

A slave ants based ant colony optimization algorithm for task scheduling in cloud computing environments

iiHadoop: an asynchronous distributed framework for incremental iterative computations

Contact Info

Product

Resources

About