Massively Parallel Processing of Whole Genome Sequence Data

Roy, Abhishek; Diao, Yanlei; Evani, Uday S.; Abhyankar, Avinash; Howarth, Clinton; Priol, Rémi Le; Bloom, Toby

doi:10.1145/3035918.3064048

Cited by 15 publications

(14 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition to enabling and evaluating horizontal scalability, the cost of an analysis and the choice of virtual machine flavors are becoming increasingly important for efficient execution of bioinformatics analysis, since pipelines are increasingly deployed and evaluated on commercial clouds [6,21,22]. However, even on dedicated clusters it is important to understand how to scale a pipeline up and out on the available resources to improve the utilization of the resources.…”

Section: Summary and Discussionmentioning

confidence: 99%

See 1 more Smart Citation

A Review of Scalable Bioinformatics Pipelines

Fjukstad

Bongo

2017

Data Sci. Eng.

View full text Add to dashboard Cite

Scalability is increasingly important for bioinformatics analysis services, since these must handle larger datasets, more jobs, and more users. The pipelines used to implement analyses must therefore scale with respect to the resources on a single compute node, the number of nodes on a cluster, and also to cost-performance. Here, we survey several scalable bioinformatics pipelines and compare their design and their use of underlying frameworks and infrastructures. We also discuss current trends for bioinformatics pipeline development.

show abstract

Section: Summary and Discussionmentioning

confidence: 99%

“…GESALL [21] is a genomic analysis platform for unmodified analysis tools that use the POSIX file system interface. An example pipeline implemented with GESALL is their implementation of the GATK variant calling reference pipeline that was used as an example in the ADAM paper [6].…”

Section: Gesall Variant Calling Pipelinementioning

confidence: 99%

A Review of Scalable Bioinformatics Pipelines

Fjukstad

Bongo

2017

Data Sci. Eng.

View full text Add to dashboard Cite

show abstract

“…The development of bioinformatics tools based on "Big-Data" technologies started in late 2008 with work on Hadoop [42], [43] and has since continued to increase [44]. Most of the advancement in this area have come in NGS data analysis, where many Hadoop-based tools have been developed [45], [46], [46]- [48] and, more recently, [13], [17]. On the other hand, there is very little on distributed streaming computing applied to the life-sciences, a part from general architecture descriptions such as [49].…”

Section: Related Workmentioning

confidence: 99%

“…Traditionally [12], the workflow steps are run on a conventional High-Performance Computing (HPC) infrastructure -a set of computing nodes accessed through a batch queuing system and equipped with a parallel shared storage system. While this is, of course, a working solution, it requires a non-trivial amount of ad-hoc manual intervention to efficiently use the available computational resources and obtain the fast turn-around times that are needed for diagnostic applications [13]. The main issues here are how to divide the work of a single job among all computing nodes and how to make the system robust to transient or permanent hardware or software failures.…”

Section: Introductionmentioning

confidence: 99%

“…As an alternative approach, there has been a recent interest [13]- [17] in supporting the NGS data processing pipelines using completely distributed platform such as Apache Hadoop. In this approach, the raw and intermediate data files are stored on a scalable file system, HDFS, and the computation is based on processing steps expressed, e.g., as Hadoop MapReduce, Spark or Flink programs.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Kafka interfaces for composable streaming genomics pipelines

Versaci¹,

Pireddu²,

Zanetti³

2017

Preprint

View full text Add to dashboard Cite

Abstract-Modern sequencing machines produce order of a terabyte of data per day, which need subsequently to go through a complex processing pipeline. The standard workflow begins with a few independent, shared-memory tools, which communicate by means of intermediate files. Given the constant increase of the amount of data produced, this approach is proving more and more unmanageable, due to its lack of robustness and scalability.In this work we propose the adoption of stream computing to simplify the genomic pipeline, boost its performance and improve its fault-tolerance. We decompose the first steps of the genomic processing in two distinct and specialized modules (preprocessing and alignment) and we loosely compose them via communication through Kafka streams, in order to allow for easy composability and integration in the already existing Hadoop-based pipelines. The proposed solution is then experimentally validated on real data and shown to scale almost linearly.

show abstract

Conceptual Modeling for Genomics: Building an Integrated Repository of Open Data

Bernasconi

Ceri

Campi

et al. 2017

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Many repositories of open data for genomics, collected by worldwide consortia, are important enablers of biological research; moreover, all experimental datasets leading to publications in genomics must be deposited to public repositories and made available to the research community. These datasets are typically used by biologists for validating or enriching their experiments; their content is documented by metadata. However, emphasis on data sharing is not matched by accuracy in data documentation; metadata are not standardized across the sources and often unstructured and incomplete. In this paper, we propose a conceptual model of genomic metadata, whose purpose is to query the underlying data sources for locating relevant experimental datasets. First, we analyze the most typical metadata attributes of genomic sources and define their semantic properties. Then, we use a top-down method for building a global-as-view integrated schema, by abstracting the most important conceptual properties of genomic sources. Finally, we describe the validation of the conceptual model by mapping it to three well-known data sources: TCGA, ENCODE, and Gene Expression Omnibus.

show abstract

Massively Parallel Processing of Whole Genome Sequence Data

Cited by 15 publications

References 26 publications

A Review of Scalable Bioinformatics Pipelines

A Review of Scalable Bioinformatics Pipelines

Kafka interfaces for composable streaming genomics pipelines

Conceptual Modeling for Genomics: Building an Integrated Repository of Open Data

Contact Info

Product

Resources

About