Advancing next‐generation sequencing data analytics with scalable distributed infrastructure

Abstract

SUMMARYWith the emergence of popular next-generation sequencing (NGS)-based genome-wide protocols such as chromatin immunoprecipitation followed by sequencing (ChIP-Seq) and RNA-Seq, there is a growing need for research and infrastructure to support the requirement of effectively analyzing NGS data. Such research and infrastructure do not replace but complement algorithmic advances developments in analyzing NGS data. We present a runtime environment, Distributed Application Runtime Environment, that supports the scalable, flexible, and extensible composition of capabilities that cover the primary requirements of NGSbased analytics. In this work, we use BFAST as a representative stand-alone tool used for NGS data analysis and a ChIP-Seq pipeline as a representative pipeline-based approach to analyze the computational requirements. We analyze the performance characteristics of BFAST and understand its dependency on different input parameters. The computational complexity of genome-wide mapping using BFAST, amongst other factors, depends upon the size of a reference genome and the data size of short reads. Characterizing the performance suggests that the mapping benefits from both scaling-up (increased fine-grained parallelism) and scaling-out (task-level parallelism -local and distributed). For certain problem instances, scaling-out can be a more efficient approach than scaling-up. On the basis of investigations using the pipeline for ChIP-Seq, we also discuss the importance of dynamical execution of tasks.

Developing eThread Pipeline Using SAGA-Pilot Abstraction for Large-Scale Structural Bioinformatics

“…It comprises high-level and easy-to-use APIs for accessing distributed resources and provisioning of job submission, monitoring, and more. It has been successfully utilized for efficient executions of loosely coupled and embarrassingly parallel applications on distributed cyberinfrastructure [ 25 , 26 ]. BigJob has three major components.…”

“…Also, cloud environments are increasingly becoming popular as a solution for massive data management, processing, and analysis [ 19 , 20 , 24 ]. Previously, SAGA-Pilot-based MapReduce and data parallelization strategies were demonstrated for life science problems, in particular, such as alignment of NGS reads [ 20 , 25 , 26 ]. Despite the successful cloud-oriented implementations of various bioinformatics tools, significantly fewer studies focused on the porting of complex structural bioinformatics algorithms to distributed computing platforms.…”

SAGA: A standardized access layer to heterogeneous Distributed Computing Infrastructure

“…RADICAL-SAGA is currently used by a wide range of projects ranging from tool developers to individual users. Illustrative but not exhaustive examples include the PANDA group (ATLAS project) [39], Gateways projects [40][41][42], as well as multiple Bioinformatics projects [43,44]. A combination of standards-based, open source and distributed development, supported by multiple independent contributors, has contributed to the sustainability of RADICAL-SAGA.…”

Special Issue for Emerging Computational Methods for the Life Sciences Workshop

“…Jha et al . presents a runtime environment, Distributed Application Runtime Environment (DARE), that supports the scalable, flexible, and extensible composition of capabilities exploring the interoperability among heterogeneously distributed computing environments for pleasingly parallel applications. DARE is a SAGA‐BigJob‐based framework motivated by the next‐generation sequencing (NGS) analysis and other similar data‐intensive applications.…”

“…Relevant contributions have been provided by Mitchel et al [1], by Lanc et al [2], by Luo et al [3], by Jha et al [4], by Ellingson et al [5], and by Schatz [6]. These contributions focus on the following:…”