Hypergraph+: An Improved Hypergraph-Based Task-Scheduling Algorithm for Massive Spatial Data Processing on Master-Slave Platforms

Abstract

Spatial data processing often requires massive datasets, and the task/data scheduling efficiency of these applications has an impact on the overall processing performance. Among the existing scheduling strategies, hypergraph-based algorithms capture the data sharing pattern in a global way and significantly reduce total communication volume. Due to heterogeneous processing platforms, however, single hypergraph partitioning for later scheduling may be not optimal. Moreover, these scheduling algorithms neglect the overlap between task execution and data transfer that could further decrease execution time. In order to address these problems, an extended hypergraph-based task-scheduling algorithm, named Hypergraph+, is proposed for massive spatial data processing. Hypergraph+ improves upon current hypergraph scheduling algorithms in two ways: (1) It takes platform heterogeneity into consideration offering a metric function to evaluate the partitioning quality in order to derive the best task/file schedule; and (2) It can maximize the overlap between communication and computation. The GridSim toolkit was used to evaluate Hypergraph+ in an IDW spatial interpolation application on heterogeneous master-slave platforms. Experiments illustrate that the proposed Hypergraph+ algorithm achieves on average a 43% smaller makespan than the original hypergraph scheduling algorithm but still preserves high scheduling efficiency.

JHTD: An Efficient Joint Scheduling Framework Based on Hypergraph for Task Placement and Data Transfer Across Geographically Distributed Data Centers

“…Yang et al improved the traditional vertex-cut hypergraph partition into a net-cut hypergraph partition [36]; it effectively balanced the partitioning results but it is not suitable for large-scale data. In addition, [37] and [38] used hypergraphs to reduce data transfer to shorten the overall makespan but they neglected the actual execution of the task. From the above work, it is evident that the deficiency of current studies is that they simply integrate the idea of the hypergraph or cannot be applied to problem optimization in geographically distributed data centers.…”

“…• Hypergraph: This algorithm assigns tasks to data centers using a hypergraph partitioning method, which evenly divides the tasks into balanced partitions according to the number of data centers. • F ast − N ewman: By considering the task scheduling problem as a community detection problem with the goal of reducing data transfer cost, this algorithm iteratively places tasks into communities while maximizing the modularity measure Q [38], where the greater modularity Q gains a better performance of community division.…”

New efficient Hadoop scheduler: Generalized particle swarm optimization and simulated annealing‐dominant resource fairness

“…The job scheduler has been developed for data locality improvement, 3,6,[8][9][10][11] improvement in energy efficiency, [12][13][14][15] job priorities, 3,16 resource demand, 10,16,17 job execution time. 9,12,[18][19][20][21] Heuristic optimization has been used to optimize the scheduling with these objectives. Many researchers have considered more than one objective.…”

“…The hypergraph is partitioned to reduce the data locality and scheduling is done based on minimum completion time. A similar yet improved hypergraph+ scheme is presented in Reference 20 in the heterogeneous environment. Another similar solution to data locality is also presented in References 27,28 with bipartite graph.…”