Memory-driven computing accelerates genomic data processing

Abstract

Becker et al. Memory-driven computing 2Next generation sequencing (NGS) is the driving force behind precision medicine and is revolutionizing most, if not all, areas of the life sciences. Particularly when targeting the major common diseases, an exponential growth of NGS data is foreseen for the next decades. This enormous increase of NGS data and the need to process the data quickly for real-world applications requires to rethink our current compute infrastructures. Here we provide evidence that memory-driven computing (MDC), a novel memory-centric hardware architecture, is an attractive alternative to current processor-centric compute infrastructures. To illustrate how MDC can change NGS data handling, we used RNA-seq assembly and pseudoalignment followed by quantification as two first examples. Adapting transcriptome assembly pipelines for MDC reduced compute time by 5.9-fold for the first step (SAMtools). Even more impressive, pseudoalignment by near-optimal probabilistic RNA-seq quantification (kallisto) was accelerated by more than two orders of magnitude with identical accuracy and indicated 66% reduced energy consumption. One billion RNA-seq reads were processed in just 92 seconds. Clearly, MDC simultaneously reduces data processing time and energy consumption. Together with the MDC-inherent solutions for local data privacy, a new compute model can be projected pushing large scale NGS data processing and primary data analytics closer to the edge by directly combining high-end sequencers with local MDC, thereby also reducing movement of large raw data to central cloud storage. We further envision that other data-rich areas will similarly benefit from this new memory-centric compute architecture.Next generation sequencing (NGS) and its manifold applications are transforming the life and medical sciences 1,2,3,4 as reflected by an exponential 860-fold increase in published data sets during the last 10 years ( Fig. 1a). Considering only human genome sequencing, the increase in genomic data over the next decade has been estimated to reach more than three Exabyte (3x10 18 byte) 5 . NGS data are projected to become on par either with data collections such as in astronomy or YouTube or the most demanding in terms of data acquisition, storage, distribution, and analysis 5 . In addition, since one of the cornerstones of genomics research and application is comparative analysis between different samples, long-term storage of genomic data will be mandatory. To cope with this data avalanche, so far, compute power is simply increased by building larger high-performance computing (HPC) clusters (more processors/cores), and by scaling out and centralizing data storage (cloud solutions) 6,7 . While currently still legitimate, it can already be foreseen that this approach is not sustainable. Centralized storage and computing leads to increased data movement, data duplication, and additional risks to data security and privacy. This is of particular interest given recent data privacy laws that have been put in place (e.g. th...

ArrowSAM: In-Memory Genomics Data Processing Using Apache Arrow

“…Other research focuses on innovative hardware platforms to execute genomics algorithms more efficiently [16]. In [17] a large pool of different types of memories are created and connected to processing resources through the Gen-Z communication protocol to investigate the concept of memory-driven computing. The memory is shared across running processes to avoid intermediate I/O operations.…”

Optimizing performance of GATK workflows using Apache Arrow In-Memory data framework

“…This systems also allows byte-addressability and load/store instructions to access memory. [49] used a Gen-Z enabled platform for genomics and reported 5.9x speedup over the SAMtools baseline implementation for a number of DNA assembly algorithms. The source code is not available.…”

ArrowSAM: In-Memory Genomics Data Processing Using Apache Arrow

“…They also compare the results on cluster to show that this platform is salable for high performance computing infrastructure and cost efficient. Memory-driven computing [25], in this research a huge pool of different types of memories created and connected to the processing resources through Gen-Z communication protocol. The memory is shared across are the running processes to avoid intermediate I/O operations.…”