An adaptive multi-agent system for task reallocation in a MapReduce job

Abstract

We study the problem of task reallocation for load-balancing of MapReduce jobs in applications that process large datasets. In this context, we propose a novel strategy based on cooperative agents used to optimise the task scheduling in a single MapReduce job. The novelty of our strategy lies in the ability of agents to identify opportunities within a current unbalanced allocation, which in turn trigger concurrent and one-to-many negotiations amongst agents to locally reallocate some of the tasks within a job. Our contribution is that tasks are reallocated according to the proximity of the resources and they are performed in accordance to the capabilities of the nodes in which agents are situated. To evaluate the adaptivity and responsiveness of our approach, we implement a prototype test-bed and conduct a vast panel of experiments in a heterogeneous environment and by exploring varying hardware configurations. This extensive experimentation reveals that our strategy significantly improves the overall runtime over the classical Hadoop data processing.

Task Bundle Delegation for Reducing the Flowtime

“…Conversely, we consider neither generalizable predictive patterns nor prior model of the data/environment are available since it is not the case for the class of practical applications we are concerned with. For instance, Baert et al have in [1] an egalitarian objective which is the minimization of the makespan (denoted C max ( − → A )). We consider here the problem of coordinating decisions between agents to find a globally optimal solution for a multi-objective function.…”

“…In our approach, a reallocation is the outcome of the negotiation process between agents adopting the same behaviour. The agent behaviour is specified in [3] by a deterministic finite state automaton 1 . An agent executes its behaviour according to its knowledge and its beliefs.…”

“…The problem of efficient task assignment among executing entities is common to many real-world applications for logistics [13,15], collective robotics [7,22], distributed systems [20,12], or more recently Big Data [1]. In particular, Data Science, which involves the processing of large volumes of data which requires distributed file systems and parallel programming, challenges distributed computing with regard to task allocation and load-balancing.…”

“…Most of these works adopt the market-oriented approach which models problems as non-cooperative games [23,7], eventually with machine learning techniques which assume past experiences [22,20]. By contrast, we suppose as [1] that: (a) the agents are cooperative, i.e. they have a partial and local perception of the task allocation but they share the same goal, and (b) neither generalizable predictive patterns nor prior model of the data/environment are available since it is not the case for the class of practical applications we are concerned with.…”

Délégation de lots de tâches pour la réduction de la durée moyenne de réalisation

“…La problématique de l'affectation efficace de tâches parmi des entités exécutantes est commune à de nombreuses applications réelles pour la logistique [16,18], la robotique collective [9,26], le calcul distribué [15,24], ou plus récemment le traitement de données massives [1]. En particulier, les sciences des données, qui exploitent de larges volumes de données sur lesquelles des calculs sont effectués en parallèle par différents noeuds, mettent à l'épreuve l'informatique distribuée en ce qui concerne l'allocation de tâches et l'équilibrage de charge.…”