Adapting bioinformatics applications for heterogeneous systems: a case study

Abstract

The advent of new sequencing technologies has generated extremely large amounts of information. To successfully apply bioinformatics tools to such large datasets, they need to exhibit scalability and ideally elasticity in diverse computing environments. We describe the application of Weaver to the PEMer structural variation detection workflow. Because the original workflow has an intractable sequential running time on large datasets, it also has a batch implementation designed for a shared file system. Using scripts provided by the developers of PEMer, along with the Weaver Python module, the Starch archive generator, and the Makeflow workflow engine, we have refactored PEMer for elastic scaling on personal clouds. Our case study describes the various challenges faced when constructing such a workflow, from dealing with failure detection, to managing dependencies, to handling the quirks of the underlying operating systems. The practice of scaling bioinformatics tools is increasingly commonplace. As such, the hands-on application of refactoring techniques to PEMer can serve as a valuable guide for those looking to reconfigure other bioinformatics software. Significantly, our customized Makeflow framework enabled elastic deployment on a wider variety of systems while substantially reducing wall clock runtimes using hundreds of cores.

Special Issue for Emerging Computational Methods for the Life Sciences Workshop

“…Lanc et al . describes the adaptation and parallelization process of Paired‐End Mapper structural variation pipeline and the Burrows–Wheeler alignment tool for execution on clusters, grids, and clouds using the weaver/starch/makeflow workflow stack. Authors describe the application of previous obtained lessons to a new workflow with and without shared file storage to tract the intractable sequential running times of these applications on large datasets.…”

Scaling up genome annotation using MAKER and work queue

“…Many of these annotation tasks have natural inherent parallelism. We previously demonstrated that a distributed computing framework can allow developers to leverage a variety of heterogeneous computational resources in a lightweight and relatively easy to program manner (Lanc et al, 2011;Moretti et al, 2009;Bui et al, 2011;Thrasher et al, 2010). There are a variety of similar existing frameworks.…”

Workshop: Opportunities and challenges of non-model ecoinformatics

“…All analysis is performed using our general and scalable framework that includes, among other tools, the discovery of SNPs [1], natural population haplotyping [2], and largescale annotation of diverse environmental samples [3,4]. We show runtime results of this framework using Amazon EC2, SGE, Condor and mixtures of the three.…”