Breaking the MapReduce stage barrier

Abstract

The MapReduce model uses a barrier between the Map and Reduce stages. This provides simplicity in both programming and implementation. However, in many situations, this barrier hurts performance because it is overly restrictive. Thus, we develop a method to break the barrier in MapReduce in a way that improves efficiency. Careful design of our barrier-less MapReduce framework results in equivalent generality and retains ease of programming. We motivate our case with, and experimentally study our barrier-less techniques in, a wide variety of MapReduce applications divided into seven classes. Our experiments show that our approach can achieve better performance times than a traditional MapReduce framework. We achieve a reduction in job completion times that is 25% on average and 87% in the best case.

MIPT: Rapid exploration and evaluation for migrating sequential algorithms to multiprocessing systems with multi-port memories

“…For example, two parents that are selected from the pool are (1,3,2,5,4,6) and (2,2,3,6,5,4). If crossover index is three, offsprings become (1, 3, 2, 6, 5, 4) and (2,2,3,5,4,6). The data dependency is checked by using the reordering rules presented in Section III-B.…”

“…Building more powerful compute units has been suggested in [2]- [4] for increasing scalability and reducing node inef-ficiency problems in future large scale processing systems. A recent publication [5] suggests more aggressive use of instruction extensible processors at compute nodes of large scale processing systems in order to tailor application for improved performance and efficiency.…”

A MapReduce scheduling algorithm for time constraints in heterogeneous environment

“…One of the most important advantages of MapReduce is its convenience, such that, programmers can process massive data without knowing the details of distributed implementation, and users can process large scale of data by only providing the Map and Reduce interface. The Map and Reduce stage is strict in the original MapReduce model, but there are some works try to break the barrier [12], [13]. The initial MapReduce model was designed for off-line data processing.…”

Analysis and acceleration of data mining algorithms on high performance reconfigurable computing platforms

“…The testing document t1 is most similar to the training document d3. performance [152]. The validity of barrier-less MapReduce model can be found in [152].…”

“…performance [152]. The validity of barrier-less MapReduce model can be found in [152]. The proof of the equality of original and pipelined MapReduce models can be found in [35].…”