Towards Efficient MapReduce Using MPI

Hoefler, Torsten; Lumsdaine, Andrew; Dongarra, Jack

doi:10.1007/978-3-642-03770-2_30

Cited by 63 publications

(32 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In fact, this kind of comparison is not completely appropriate. Recent research has shown how the MapReduce approach can be implemented using corresponding functions in the MPI protocol (Hoefler et al 2009). Our principal goal here is not to advocate the specific MapReduce implementation we have used here: rather it is to emphasise that several kinds of parallelism can be achieved by definitions of two simple functions.…”

Section: Discussionmentioning

confidence: 99%

Data and task parallelism in ILP using MapReduce

2011

View full text Add to dashboard Cite

Nearly two decades of research in the area of Inductive Logic Programming (ILP) have seen steady progress in clarifying its theoretical foundations and regular demonstrations of its applicability to complex problems in very diverse domains. These results are necessary, but not sufficient, for ILP to be adopted as a tool for data analysis in an era of very large machine-generated scientific and industrial datasets, accompanied by programs that provide ready access to complex relational information in machine-readable forms (ontologies, parsers, and so on). Besides the usual issues about the ease of use, ILP is now confronted with questions of implementation. We are concerned here with two of these, namely: can an ILP system construct models efficiently when (a) Dataset sizes are too large to fit in the memory of a single machine; and (b) Search space sizes becomes prohibitively large to explore using a single machine. In this paper, we examine the applicability to ILP of a popular distributed computing approach that provides a uniform way for performing data and task parallel computations in ILP. The MapReduce programming model allows, in principle, very large numbers of processors to be used without any special understanding of the underlying hardware or software involved. Specifically, we show how the MapReduce approach can be used to perform the coverage-test that is at the heart of many ILP systems, and to perform multiple searches required by a greedy set-covering algorithm used by some popular ILP systems. Our principal findings with synthetic and real-world datasets for both data and task parallelism are these: (a) Ignoring overheads, the time to perform the computations concurrently increases with the size of the dataset for data parallelism and with the size of the search space for task parallelism. For data parallelism this increase is roughly in proportion to increases in dataset size; (b) If a MapReduce implementation is used as part of an ILP system, then benefits for data parallelism can only be expected above some minimal Mach Learn (2012) 86:141-168 dataset size, and for task parallelism can only be expected above some minimal search-space size; and (c) The MapReduce approach appears better suited to exploit data-parallelism in ILP.

show abstract

Section: Discussionmentioning

confidence: 99%

Data and task parallelism in ILP using MapReduce

2011

View full text Add to dashboard Cite

show abstract

“…In terms of scheduling, literature [22] tried to use a priority-based scheduling strategy to improve efficiency of MapReduce. Literature [23] proposed the MapReduce optimized implementation based on MPI, using MPI-3 new features such as MPI Reduce Local to get 25% of the performance on the cluster of 127 nodes. Purdue University [24] researchers take the method of hunger-by loosening the synchronization requirements of schedule (eager scheduling) to improve efficiency of the MapReduce task [25] .Barcelona Supercomputer Center and researchers at the IBM Watson laboratory research scheduling strategy [26], with a view to improve performance.…”

Section: Summarization and Prospectmentioning

confidence: 99%

Research on Hybrid Distributed Computing System Based on Embedded System

Li¹,

Mu²,

Zhang³

et al. 2016

IJGDC

View full text Add to dashboard Cite

show abstract

“…The power offered to users by this abstraction has advocated new approaches at solving large-scale problems in industrial settings [8]. There are also systems that have implemented MapReduce on top of MPI [13,22] as well as multi-GPU architectures [25].…”

Section: Simplified Large-scale Data Processingmentioning

confidence: 99%

Simplified parallel domain traversal

Kendall

Wang

Allen

et al. 2011

Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

Many data-intensive scientific analysis techniques require global domain traversal, which over the years has been a bottleneck for efficient parallelization across distributedmemory architectures. Inspired by MapReduce and other simplified parallel programming approaches, we have designed DStep, a flexible system that greatly simplifies efficient parallelization of domain traversal techniques at scale. In order to deliver both simplicity to users as well as scalability on HPC platforms, we introduce a novel two-tiered communication architecture for managing and exploiting asynchronous communication loads. We also integrate our design with advanced parallel I/O techniques that operate directly on native simulation output. We demonstrate DStep by performing teleconnection analysis across ensemble runs of terascale atmospheric CO2 and climate data, and we show scalability results on up to 65,536 IBM BlueGene/P cores.

show abstract

Towards Efficient MapReduce Using MPI

Cited by 63 publications

References 20 publications

Data and task parallelism in ILP using MapReduce

Data and task parallelism in ILP using MapReduce

Research on Hybrid Distributed Computing System Based on Embedded System

Simplified parallel domain traversal

Contact Info

Product

Resources

About