VC@Scale: Scalable and high-performance variant calling on cluster environments

Abstract

Background Recently many new deep learning–based variant-calling methods like DeepVariant have emerged as more accurate compared with conventional variant-calling algorithms such as GATK HaplotypeCaller, Sterlka2, and Freebayes albeit at higher computational costs. Therefore, there is a need for more scalable and higher performance workflows of these deep learning methods. Almost all existing cluster-scaled variant-calling workflows that use Apache Spark/Hadoop as big data frameworks loosely integrate existing single-node pre-processing and variant-calling applications. Using Apache Spark just for distributing/scheduling data among loosely coupled applications or using I/O-based storage for storing the output of intermediate applications does not exploit the full benefit of Apache Spark in-memory processing. To achieve this, we propose a native Spark-based workflow that uses Python and Apache Arrow to enable efficient transfer of data between different workflow stages. This benefits from the ease of programmability of Python and the high efficiency of Arrow’s columnar in-memory data transformations. Results Here we present a scalable, parallel, and efficient implementation of next-generation sequencing data pre-processing and variant-calling workflows. Our design tightly integrates most pre-processing workflow stages, using Spark built-in functions to sort reads by coordinates and mark duplicates efficiently. Our approach outperforms state-of-the-art implementations by >2 times for the pre-processing stages, creating a scalable and high-performance solution for DeepVariant for both CPU-only and CPU + GPU clusters. Conclusions We show the feasibility and easy scalability of our approach to achieve high performance and efficient resource utilization for variant-calling analysis on high-performance computing clusters using the standardized Apache Arrow data representations. All codes, scripts, and configurations used to run our implementations are publicly available and open sourced; see https://github.com/abs-tudelft/variant-calling-at-scale.

Framing Apache Spark in life sciences

“…The table shows that apart from some tools that reports tests only on a multi-core workstation ( [16] , [17] , [18] , [19] ), Spark has been widely used to implement tools aimed at parallelizing the computation on a distributed computing environment. Most of these tools have been specifically devised for, or tested on, a cloud environment ( [20] , [21] , [22] , [23] , [24] , [25] , [26] , [27] , [28] [29] , [30] , [31] , [32] [33] , [34] , [35] , [36] , [37] ). Being the increasing availability of IaaS (Infrastructure as a Service) cloud computing services, it is desirable that the released tools are commonly designed to be supported also by such infrastructures.…”

“…CMAN Ext. tools/frameworks Genomics genome assembly SORA [20] de novo genome assembly GraphX ✓ - ✓ - - - variant calling DECA [21] copy number variantion discovery MLlib ✓ - ✓ - - ADAM ADS-HCSpark [48] SNPs and indels calling - ✓ - - - - - SparkGA2 [22] variant calling - ✓ - ✓ ✓ - - SparkRA [49] GATK best-practices pipeline - ✓ - - - - - DeepVariant on Spark [23] SNPs and indels calling - ✓ ✓ ✓ ✓ - Apache Parquet VC@Scale [24] SNPs and indels calling - ✓ ✓ ✓ - - Apache Arrow Halvade Somatic [25] somatic variant calling - ✓ - ✓ - - - …”

“…It should be pointed out that some applications in the fields of genomics ( [23] , [24] ) and biomedicine ( [35] , [36] ) have also been designed to distribute the computation on multiple GPUs on multiple nodes. GPU technology is widely used in life science.…”

GenMPI: Cluster Scalable Variant Calling for Short/Long Reads Sequencing Data

“…Data formats like Apache Parquet, Apache Arrow, Apache Avro have been explored extensively in conjunction with these frameworks to store and process genomic data efficiently. These frameworks include ADAM (Massie et al, 2013), SparkGA2 (Mushtaq et al, 2019), VC@Scale (Ahmad et al, 2021) and Halvade (Decap et al, 2015). Due to many underlying dependencies, inefficient memory usage, issues related to scalability, cluster deployment challenges as well as incompatible data formats, solutions based on these frameworks are still not widely used in the mainstream Bioinformatics community.…”

Cloud-native distributed genomic pileup operations

“…In the recent Genome Analysis Toolkit (GATK, McKenna et al ., 2010) version, several programs (including pileup calculations) have been implemented in a distributed manner ready to be run on the Apache Spark cluster. Other research studies confirm that big data programming paradigms can be successfully applied to many genomics analyses (Guo et al ., 2018, Capuccini et al ., 2020, Wiewiórka et al ., 2018, Wiewiórka et al ., 2017) including variant calling(Ahmad et al ., 2021). The analysis of the ever-increasing genomic data sets involves significant financial investments and administrative efforts to maintain secure and fault-tolerant storage solutions as well as fast and scalable processing units.…”