Data Redistribution Algorithms for Heterogeneous Processor Rings

Abstract

We consider the problem of redistributing data on homogeneous and heterogeneous ring of processors. The problem arises in several applications, each time after that a load-balancing mechanism is invoked (but we do not discuss the load-balancing mechanism itself). We provide algorithms that aim at optimizing the data redistribution, both for uni-directional and bi-directional rings, and we give complete proofs of correctness. One major contribution of the paper is that we are able to prove the optimality of the proposed algorithms in all cases except that of a bi-directional heterogeneous ring, for which the problem remains open. Keywords: Heterogeneous rings, data redistribution algorithms, load-balancing RésuméDans ce rapport, nous nous intéressons au problème de redistribution de données sur des anneaux de processeurs homogènes et hétérogènes. Ce problème surgit dans plusieurs applications, après chaque phase d'équilibrage de charge (nous ne discutons pas ici du mécanisme d'équilibrage de charge lui-même). Nous proposons des algorithmes qui visentà optimiser la redistribution de données pour des anneaux unidirectionnels et bidirectionnels, et nous donnons toutes les preuves de correction de ces algorithmes. Une des contributions principales de ce rapport est que nous pouvons prouver l'optimialité des algorithmes proposés dans tous les cas, sauf dans le cas d'un anneau hétérogène bidirectionnel, pour lequel le problème reste ouvert.

Optimizing and Tuning MapReduce Jobs to Improve the Large-Scale Data Analysis Process

“…Furthermore, the overhead of transferring unprocessed data from slow nodes to fast ones is high based on the large volume of data to be moved. To solve this limitation and enhance the MapReduce performance in a cluster environment, we extended the data redistribution algorithm, which aimed to partition a large data set into small fragments being distributed across multiple nodes in a cluster that arises due to dynamic data insertions and deletions. Approach 2: The Number of Map and Reduce Tasks for a Job May Cause Performance Problems in Map Reduce . The dependence among reduce and map tasks can slow down the performance of clusters by an imbalanced workload, while some nodes are underutilized and others are overloaded.…”

“…However, MapReduce consists of different interleaving stages, each requiring different I/O workloads and patterns. A novel approach was taken by Blanas et al for adaptively tuning the disk pairs’ schedulers in both the hypervisor and the virtual machines during the execution of a single MapReduce job and compare the performance improvement on a sort benchmark with Hadoop to achieve the shortest execution time of MapReduce using various parameters proposed a cost model to predict the total execution time of jobs and their optimal assignments, and Scheduling Algorithm MapReduce (SAMR) proposed a dynamic task calculate process that adapts to the continuously varying environment automatically. One of the most important requirements for effective performance tuning is to discover those important parameters that are related to tuning a job for all features.…”

Scheduling and Data Redistribution Strategies on Tree Platforms

“…Lots of important projects [6,7] involves splitting load into identical and independent tasks. Second, some researchers design algorithms for certain particular cases [3], or restrict the platform architecture [4]. Finally, data redistribution is designed to equilibrate finishing times and load.…”

“…The existing algorithms fail to consider either unbalanced load on workers, or the computation phase in optimizations. In addition, some of existing algorithms suffer from restriction to the platforms of certain type, for example, Moore Based Binary-Search Algorithm (MBBSA) on star platforms [2] and redistribution algorithms on ring platforms [3]. Despite the existence of redistribution algorithms on tree platforms, communication and computation time were neglected, for instance, M.Y.Wu [4].…”

Scheduling and Data Redistribution Strategies on Star Platforms

“…Unfortunately already simple redistribution problems are NP complete [8]. For this reason, optimal algorithms can be designed only for particular cases, as it is done in [13]. In their research, the authors restrict the platform architecture to ring topologies, both uni-directional and bidirectional.…”