Pairwise Element Computation with MapReduce

Kiefer, Tim; Volk, Peter Benjamin; Lehner, Wolfgang

doi:10.1145/1851476.1851595

Cited by 11 publications

(9 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To the best of our knowledge, in the literature bijective functions have not ever been proposed for workload balancing in symmetric all-pairs computation. In [18], the authors used a very similar job numbering approach to ours in this study, but did not derive a bijective function for symmetric all-pairs computation. Our framework can be applied to cases with identical (e.g.…”

Section: ) Non-symmetric All-pairs Computationmentioning

confidence: 99%

Parallel Pairwise Correlation Computation on Intel Xeon Phi Clusters

Liu

Pan

Aluru

2016

2016 28th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD)

View full text Add to dashboard Cite

Co-expression network is a critical technique for the identification of inter-gene interactions, which usually relies on all-pairs correlation (or similar measure) computation between gene expression profiles across multiple samples. Pearson's correlation coefficient (PCC) is one widely used technique for gene co-expression network construction. However, all-pairs PCC computation is computationally demanding for large numbers of gene expression profiles, thus motivating our acceleration of its execution using high-performance computing. In this paper, we present LightPCC, the first parallel and distributed allpairs PCC computation on Intel Xeon Phi (Phi) clusters. It achieves high speed by exploring the SIMD-instruction-leveland thread-level parallelism within Phis as well as acceleratorlevel parallelism among multiple Phis. To facilitate balanced workload distribution, we have proposed a general framework for symmetric all-pairs computation by building bijective functions between job identifier and coordinate space for the first time. We have evaluated LightPCC and compared it to two CPU-based counterparts: a sequential C++ implementation in ALGLIB and an implementation based on a parallel general matrix-matrix multiplication routine in Intel Math Kernel Library (MKL) (all use double precision), using a set of gene expression datasets. Performance evaluation revealed that with one 5110P Phi and 16 Phis, LightPCC runs up to 20.6× and 218.2× faster than ALGLIB, and up to 6.8× and 71.4× faster than single-threaded MKL, respectively. In addition, LightPCC demonstrated good parallel scalability in terms of number of Phis. Source code of LightPCC is publicly available at http://lightpcc.sourceforge.net.

show abstract

Section: ) Non-symmetric All-pairs Computationmentioning

confidence: 99%

Parallel Pairwise Correlation Computation on Intel Xeon Phi Clusters

Liu

Pan

Aluru

2016

2016 28th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD)

View full text Add to dashboard Cite

show abstract

“…Another variant that has received attention is computing pairwise "functional" computations (not necessarily similarities) between elements in large datasets [23,15].…”

Section: Related Workmentioning

confidence: 99%

“…The reducer iterates over the signatures and keeps the last B in the queue (lines [13][14][15][16][17]. At each iteration, the current signature is compared to all signatures in the queue, and hamming distances are emitted as output (lines 8-12).…”

Section: Basic Algorithmmentioning

confidence: 99%

“…Note that some comparisons will be missed at the boundary of chunks (e.g., the last element of a chunk is not compared to the first element of the next chunk). We solve this problem by appending the last B signatures of the previous chunk at the beginning of the current chunk ( Figure 3, lines [15][16]. This results in redundancy in the chunks emitted to disk (the last B signatures of a chunk are the same as the first B signatures of the next one), but the redundancy is negligible especially since B M .…”

Section: Improved Algorithmmentioning

confidence: 99%

See 1 more Smart Citation

No free lunch

Ture

Elsayed

Lin

2011

Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

This work explores the problem of cross-lingual pairwise similarity, where the task is to extract similar pairs of documents across two different languages. Solutions to this problem are of general interest for text mining in the multilingual context and have specific applications in statistical machine translation. Our approach takes advantage of cross-language information retrieval (CLIR) techniques to project feature vectors from one language into another, and then uses locality-sensitive hashing (LSH) to extract similar pairs. We show that effective cross-lingual pairwise similarity requires working with similarity thresholds that are much lower than in typical monolingual applications, making the problem quite challenging. We present a parallel, scalable MapReduce implementation of the sort-based sliding window algorithm, which is compared to a brute-force approach on German and English Wikipedia collections. Our central finding can be summarized as "no free lunch": there is no single optimal solution. Instead, we characterize effectivenessefficiency tradeoffs in the solution space, which can guide the developer to locate a desirable operating point based on application-and resource-specific constraints.

show abstract

“…While some are applicable only to specific problems (Parikshit et al 2009;Cormen et al 1990;Wauthier et al 2013;Krejčí 2018), a general strategy that can be applied across multiple applications is to parallelize the comparisons, hopefully in some kind of efficient manner. Kiefer et al (Kiefer et al 2010) reviewed three ways to accomplish this. The first ("broadcast") approach consists solely in having multiple processes performing comparisons each on a subset of pairs.…”

Section: Introductionmentioning

confidence: 99%

Novel Approach for Parallelizing Pairwise Comparison Problems as Applied to Detecting Segments Identical By Decent in Whole-Genome Data

Sapin

2020

Preprint

View full text Add to dashboard Cite

AbstractMotivationPairwise comparison problems arise in many areas of science. In genomics, datasets are already large and getting larger, and so operations that require pairwise comparisons—either on pairs of SNPs or pairs of individuals—are extremely computationally challenging. We propose a generic algorithm for addressing pairwise comparison problems that breaks a large problem (of order n2 comparisons) into multiple smaller ones (each of order n comparisons), allowing for massive parallelization.ResultsWe demonstrated that this procedure is very efficient for calling identical by descent (IBD) segments between all pairs of individuals in the UK Biobank dataset, with a user time savings roughly 180-fold over the traditional (non-parallel) approach to detecting such segments. This efficiency should extend to other methods of IBD calling and, more generally, to other pairwise comparison tasks in genomics or other areas of science.Contactemmanuel.sapin@colorado.edu

show abstract

Pairwise Element Computation with MapReduce

Cited by 11 publications

References 7 publications

Parallel Pairwise Correlation Computation on Intel Xeon Phi Clusters

Parallel Pairwise Correlation Computation on Intel Xeon Phi Clusters

No free lunch

Novel Approach for Parallelizing Pairwise Comparison Problems as Applied to Detecting Segments Identical By Decent in Whole-Genome Data

Contact Info

Product

Resources

About