An OpenMP algorithm and implementation for clustering biological graphs

Abstract

Graph algorithms on parallel architectures present an interesting case study for irregular applications. Among the graph algorithms popular in scientific computing, graph clustering or community detection has numerous applications in computational biology. However, this operation also poses serious computational challenges because of irregular memory access patterns, large memory requirements, and their dependence on other auxiliary (also irregular) data structures to supplement processing. In this paper, we address the problem of graph clustering on shared memory machines. We present a new OpenMP-based parallel algorithm called pClust-sm, which uses adjacency lists, hash tables and unionfind data structures in parallel. The algorithm improves both the asymptotic runtime and memory complexities of a previous serial implementation. Preliminary results show that this algorithm can scale up to 8 threads (cores) of a shared memory machine on a real world metagenomics input graph with 1.2M vertices and 100M edges. More importantly, the new implementation drastically reduces the time to solution from the order of several hours to just over 4 minutes, and in addition, it enhances the problem size reach by at least one order of magnitude.

Novel Approach for Parallelizing Pairwise Comparison Problems as Applied to Detecting Segments Identical By Decent in Whole-Genome Data

“…Situations where all pairs of observations in a dataset must be compared arise in many areas of science. For example, in protein studies, forming the graphs used in protein clustering relies on finding a protein's likeness to every other protein (Chapman and Kalyanaraman 2011;Sapin et al 2016), and in physics, calculating the total force each body has on every other body is required in order to predict the position and motion of all bodies in the n-body problem (Leimanis and Minorsky 1958). Such pairwise comparison problems are extremely computationally challenging with large datasets because they grow at the square of the sample size (are of order O(n 2 )).…”

Scaling Distributed All-Pairs Algorithms

“…In biometrics applications, a similarity matrix can be formed using a set of images compared with itself using facial recognition [2]. In metagenomics, finding a protein's likeness to every other protein is a crucial part of forming the complex graphs used in protein clustering, which has led to new discoveries of protein functions [3].…”

Efficient computation and communication management for all-pairs interactions

“…In biometrics applications, a similarity matrix can be formed using a set of images compared with itself using facial recognition [Phillips et al (2005)]. In metagenomics, finding a protein's likeness to every other protein is a crucial part of forming the complex graphs used in protein clustering, which has led to new discoveries of protein functions [Chapman and Kalyanaraman (2011)].…”