Phoenix rebirth: Scalable MapReduce on a large-scale shared-memory system

Yoo, Richard M.; Romano, Anthony G.; Kozyrakis, Christos

doi:10.1109/iiswc.2009.5306783

Cited by 206 publications

(132 citation statements)

References 11 publications

Supporting

Mentioning

129

Contrasting

Unclassified

Order By: Relevance

“…Cilk offers a simple means to express parallel loops (using the cilk_for syntax) and reductions (using generalized reducer hyperobjects [11]). These two concepts have the same conceptual programming complexity as other map-reduce systems [26], [27], [31], [33], [35]. CilkMR, however, retains the structure of the sequential code, which is in stark contrast to previously proposed map-reduce frameworks.…”

Section: Introductioncontrasting

confidence: 42%

“…Yoo et al [35] improved the scalability of Phoenix for a 256-thread SPARC T2 machine. They found that the internal data structures that store intermediate data are critical for performance.…”

Section: Introductionmentioning

confidence: 99%

“…Data movement is vertical between levels of the memory hierarchy instead of horizontal between compute nodes. Load balancing can be achieved at a much finer granularity of work items and shared memory map-reduce runtime systems do not usually provide fault tolerance [26], [27], [31], [33], [35].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A scalable and composable map-reduce system

Arif

Vandierendonck

Nikolopoulos

et al. 2016

2016 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

Section: Introductioncontrasting

confidence: 42%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A scalable and composable map-reduce system

Arif

Vandierendonck

Nikolopoulos

et al. 2016

2016 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

“…In alternative hardware environments, such as a shared-memory multi-core architecture, the underlying storage may support constant-time access. There are MapReduce implementations that target such environments, such as Phoenix [47,48], which is a shared-memory implementation of MapReduce for multi-core environments, and MARS [49,50], which accelerates MapReduce using graphics processors for coprocessing. In a similar vein, the Nornir run-time [51] offers an efficient multicore implementation of Kahn process networks [52], allowing the structuring of CPU-intensive computations as arbitrary communication graphs, which may contain cycles.…”

Section: Alternative and Hybrid Architecturesmentioning

confidence: 99%

Cogset: a high performance MapReduce engine

Valvåg

Johansen

Kvalnes

2012

Concurrency and Computation

View full text Add to dashboard Cite

MapReduce has become a widely employed programming model for large-scale data-intensive computations. Traditional MapReduce engines employ dynamic routing of data as a core mechanism for fault tolerance and load balancing. An alternative mechanism is static routing, which reduces the need to store temporary copies of intermediate data, but requires a tighter coupling between the components for storage and processing. The initial intuition motivating our work is that reading and writing less temporary data could improve performance, while the tight coupling of storage and processing could be leveraged to improve data locality.We therefore conjecture that a high-performance MapReduce engine can be based on static routing, while preserving the non-functional properties associated with traditional engines. To investigate this thesis, we design, implement, and experiment with Cogset, a distributed MapReduce engine that deviates considerably from the traditional design.We evaluate the performance of Cogset by comparing it to a widely used traditional MapReduce engine using a previously established benchmark. The results confirm our thesis that a high-performance MapReduce engine can be based on static routing, although analysis indicates that the reasons for Cogset's performance improvements are more subtle than expected. Through our work we develop a better understanding of static routing, its benefits and limitations, and its ramifications for a MapReduce engine.A secondary goal of our work is to explore how higher-level abstractions that are commonly built on top of MapReduce will interact with an execution engine based on static routing. Cogset is therefore designed with a generic, low-level core interface, upon which MapReduce is implemented as a relatively thin layer, as one of several supported programming interfaces.At its core, Cogset provides a few fundamental mechanisms for reliable and distributed storage of data, and parallel processing of statically partitioned data. While this dissertation mainly focuses on how these capabilities are leveraged to implement a distributed MapReduce engine, we also demonstrate how two other higher-level abstractions were built on top of Cogset. These may serve as alternative access points for data-intensive applications, and illustrate how some of the lessons learned from Cogset can be applicable in a broader context.

show abstract

“…Ten annotations are used to specify concurrency and synchronization; five are used to program MapReduce [22,61] computations.…”

Section: The Gossamer Approach and Contributionsmentioning

confidence: 99%