PipeMEM: A Framework to Speed Up BWA-MEM in Spark with Low Overhead

Abstract

(1) Background: DNA sequence alignment process is an essential step in genome analysis. BWA-MEM has been a prevalent single-node tool in genome alignment because of its high speed and accuracy. The exponentially generated genome data requiring a multi-node solution to handle large volumes of data currently remains a challenge. Spark is a ubiquitous big data platform that has been exploited to assist genome alignment in handling this challenge. Nonetheless, existing works that utilize Spark to optimize BWA-MEM suffer from higher overhead. (2) Methods: In this paper, we presented PipeMEM, a framework to accelerate BWA-MEM with lower overhead with the help of the pipe operation in Spark. We additionally proposed to use a pipeline structure and in-memory-computation to accelerate PipeMEM. (3) Results: Our experiments showed that, on paired-end alignment tasks, our framework had low overhead. In a multi-node environment, our framework, on average, was 2.27× faster compared with BWASpark (an alignment tool in Genome Analysis Toolkit (GATK)), and 2.33× faster compared with SparkBWA. (4) Conclusions: PipeMEM could accelerate BWA-MEM in the Spark environment with high performance and low overhead.

VC@Scale: Scalable and high-performance variant calling on cluster environments

“…Almost all BWA-MEM cluster-scaled implementations (SparkBWA [ 8 ], BWASpark [ 9 ], PipeMEM [ 10 ], ADAM [ 7 ], and SparkGA2 [ 6 ]) run multiple BWA-MEM instances on each Spark worker node as Spark tasks, which degrades the underlying efficient single-node multi-threaded scalability of this tool. Instead we use 1 BWA-MEM instance on each Spark worker node, storing output SAM files on storage and merging these SAM files to generate a single output SAM file.…”

“…pBWA [ 30 ] and mpiBLAST [ 31 ] use MPI, and CUSHAW3 [ 32 ] uses UPC++. Similarly ADAM’s Cannoli [ 7 ], SparkBWA [ 8 ], and PipeMEM [ 10 ] are a few Apache Spark–based BWA implementations that use BWA as loosely integrated underneath these implementations while GATK BWASpark modifies the original BWA to exploit the Spark scheduling and shuffling functionality to run BWA instances in parallel on clusters.…”

“…Spark commonly uses HDFS to read/write data but also supports other storage systems such as Network File System (NFS), HBase, and Amazon’s S3. Many variant-calling workflows and tools have been developed over the past decade since its first release, including SparkGA2 [ 6 ], ADAM [ 7 ], SparkBWA [ 8 ], BWASpark [ 9 ], PipeBWA [ 10 ], and others.…”

Big Data in metagenomics: Apache Spark vs MPI

“…Recently, Big Data technologies such as Apache Hadoop [4] and Apache Spark [5,6] are being employed. They allow the usage of high-level programming languages, such as Java, Python, or Scala, while providing ease of use and performance [7][8][9][10][11].…”

“…Big Data technologies, on the other hand, have become increasingly popular, and their usage is not longer restricted to data analytics, but has been successfully used in fields like bioinformatics [7][8][9][10][11]15], chemistry [29,30], or medicine [31,32]. Technologies like Apache Hadoop [4] or Apache Spark [5] offer a scalable way to process enormous amounts of data in large clusters of "cheap" computers or virtual machines in the cloud, using simple programming models.…”