Scheduling in a dynamic heterogeneous distributed system using estimation error

Abstract

a b s t r a c tIn real-world dynamic heterogeneous distributed systems, allocating tasks to processors can be an inefficient process, due to the dynamic nature of the resources, and the tasks to be processed. The information about these tasks and resources is not known a priori, and thus must be estimated online. We utilize the accuracy of these estimates, and when combined with different objectives, such as minimizing makespan and evenly distributing load, naturally gives rise to a family of four different scheduling algorithms. The algorithms have been implemented on a real-world heterogeneous distributed system with up to 90 processors. A set of real-world problems from the areas of cryptography, bioinformatics, and biomedical engineering were used as a test-set to measure the effectiveness of the scheduling algorithms. We have found that considering estimation error when allocating tasks to processors can provide more efficient solutions, than when estimation error is not considered. We have found that using a simple heuristic, combined with estimation error, can in some cases provide solutions approaching the efficiency of complicated well-known evolutionary algorithms.

Adaptive online scheduling of tasks with anytime property on heterogeneous resources

“…However, these methods cannot be used if the processing times of the tasks are significantly different. A more successful approach is to use statistical methods or methods from machine learning [25,15]. In comparison with single resource environment, estimation of the processing time of the tasks in heterogeneous systems is even more complicated, i.e.…”

“…In [17], the benchmark result is appended to a feature vector of a task and the k-nearest neighbor method is used to find similar observations from which an estimation of the processing time is computed. An alternative approach to appending a benchmark result to a feature vector is to scale the processing time of a task by the benchmark result of the resource on which the task is processed [25]. However, none of these works deal with scheduling of tasks solved by anytime algorithms.…”

MOMC: Multi-objective and Multi-constrained Scheduling Algorithm of Many Tasks in Hadoop

“…Since the time and the hardware are two valuable resources, it is very important to schedule the data processing very effective. This is way the algorithms behind the scheduler are an important topic for parallel and distributed system [1].…”