A robust main-memory compression scheme

Ekman, Magnus; Stenström, Per

doi:10.1109/isca.2005.6

Cited by 147 publications

(132 citation statements)

References 19 publications

Supporting

Mentioning

128

Contrasting

Unclassified

Order By: Relevance

“…A node can also prefetch the input for its next Map or Reduce task while processing the current one, which is similar to the double-buffering schemes used in streaming models [23]. Bandwidth and cache space can be preserved using hardware compression of intermediate pairs which tend to have high redundancy [10].…”

Section: Runtime Systemmentioning

confidence: 99%

“…The runtime can also provide cache replacement hints for input and output pairs accessed in Map and Reduce tasks [25]. Finally, hardware compression/decompression of intermediate outputs as they are emitted in the Map stage or consumed in the Reduce stage can reduce bandwdith and storage requirements [10]. This section describes the experimental methodology we used to evaluate Phoenix.…”

Section: Concurrency and Locality Managementmentioning

confidence: 99%

See 1 more Smart Citation

Evaluating MapReduce for Multi-core and Multiprocessor Systems

Ranger

Raghuraman

Penmetsa

et al. 2007

2007 IEEE 13th International Symposium on High Performance Computer Architecture

769

447

View full text Add to dashboard Cite

This paper evaluates the suitability of the MapReduce model for multi-core and multi-processor systems. MapReduce was created by Google for application development on data-centers with thousands of servers. It allows programmers to write functional-style code that is automatically parallelized and scheduled in a distributed system.We describe Phoenix, an implementation of MapReduce for shared-memory systems that includes a programming API and an efficient runtime system. The Phoenix runtime automatically manages thread creation, dynamic task scheduling, data partitioning, and fault tolerance across processor nodes. We study Phoenix with multi-core and symmetric multiprocessor systems and evaluate its performance potential and error recovery features. We also compare MapReduce code to code written in lower-level APIs such as P-threads. Overall, we establish that, given a careful implementation, MapReduce is a promising model for scalable performance on shared-memory systems with simple parallel code.

show abstract

Section: Runtime Systemmentioning

confidence: 99%

Section: Concurrency and Locality Managementmentioning

confidence: 99%

Evaluating MapReduce for Multi-core and Multiprocessor Systems

Ranger

Raghuraman

Penmetsa

et al. 2007

2007 IEEE 13th International Symposium on High Performance Computer Architecture

769

447

View full text Add to dashboard Cite

show abstract

“…The operating system running on the core uses part of this memory, while the user can use the rest. Intel provides a custom Linux kernel that during the boot process, allocates 5 (34)(35)(36)(37)(38) map configuration registers of cores. Entry 250 addresses the system interface; access to this memory is confined to the PCIe driver.…”

Section: Scc Address Spacesmentioning

confidence: 99%

Hybrid address spaces: A methodology for implementing scalable high-level programming models on non-coherent many-core architectures

Papagiannis

Nikolopoulos

2014

Journal of Systems and Software

View full text Add to dashboard Cite

This paper introduces hybrid address spaces as a fundamental design methodology for implementing scalable runtime systems on many-core architectures without hardware support for cache coherence. We use hybrid address spaces for an implementation of MapReduce, a programming model for large-scale data processing, and the implementation of a remote memory access (RMA) model. Both implementations are available on the Intel SCC and are portable to similar architectures. We present the design and implementation of HyMR, a MapReduce runtime system whereby different stages and the synchronization operations between them alternate between a distributed memory address space and a shared memory address space, to improve performance and scalability. We compare HyMR to a reference implementation and we find that HyMR improves performance by a factor of 1.71× over a set of representative MapReduce benchmarks. We also compare HyMR with Phoenix++, a state-of-art implementation for systems with hardware-managed cache coherence in terms of scalability and sustained to peak data processing bandwidth, where HyMR demonstrates improvements of a factor of 3.1× and 3.2× respectively. We further evaluate our hybrid remote memory access (HyRMA) programming model and assess its performance to be superior of that of message passing.

show abstract

“…Several proposals have been made suggesting memory compression to save bandwidth [2][15] [13]. We deem these approaches orthogonal to fine-grained fetch since compression of sub-blocks could further alleviate the bandwidth problem.…”

Section: Related Workmentioning

confidence: 99%